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SECTION  1 - INTRODUCTION 


1 .1  PURPOSE 

Several  Air  Force  regulations,  specifications,  and  standards  provide  guidance 
on  the  conduct  of  reviews  and  audits  in  the  acquisition  of  systems  and  compu- 
ter program  configuration  items.  This  guidebook  (1)  combines  existing  guid- 
ance regarding  reviews  and  audits  currently  contained  in  a number  of  different 
official  documents  into  a single  document,  and  (2)  narrows  the  focus  of  exist- 
ing guidance  to  those  problems  inherent  in  software  acquisition  management. 
Where  appropriate,  existing  guidance  is  extended  to  include  practices  and  pro- 
cedures based  on  the  practical  experience  of  the  author  and  others  in  acquir- 
ing software.  The  objective  of  this  document  is  to  instruct  Air  Force  Program 
Office  personnel,  and  an  evolving  position  within  the  Engineering  Division 
(referred  to  herein  as  the  Software  Director),  in  the  effective  use  of  reviews 
and  audits  as  management  tools  for  the  acquisition  of  software. 

1 .2  SCOPE 

The  scope  of  this  guidebook  is  limited  to  engineering  design  reviews  and 
configuration  management  audits  related  to  the  acquisition  of  computer 
program  configuration  items  (CPCls)  procured  as  oart  of  a command,  control, 
and  communications  (C3)  system.  Detailed  guidance  is  provided  for  the  follow- 
ing formal  reviews  and  audits: 

• System  Requirements  Review  (SRR) 

• System  Design  Review  (SDR) 

• Preliminary  Design  Review  (PDR) 

• Critical  Design  Review  (CDR) 

e Functional  Configuration  Audit  (FCA) 

• Physical  Configuration  Audit  (PCA) 

• Formal  Qualification  Review  (FQR) 
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1 . 3 PURPOSE  OF  ENGINEERING  DESIGN  REVIEWS 

Engineering  design  reviews  provide  the  Program  Office  (PO)  with  a tool  for 
monitoring  the  developing  organization's  technical  progress.  Design  reviews 
are  milestones  between  baselines  which  provide  visibility  into  the  technical 
progress  of  a particular  phase  within  the  system  acquisition  life  cycle.  A 
baseline  is  a point  of  control  normally  defined  by  an  approved  contract  speci- 
fication. Normally,  design  reviews  are  phase  oriented  and  directly  related  to 
the  specific  level  of  detail  available  at  the  time  of  the  review.  The  SRR  and 
SDR  are  normally  Validation  Phase  reviews  and  are  used  to  monitor  the 
progress  of  detailing  the  system/segment  requirements  and  allocating  these 
requirements  to  configuration  items  (CIs).  The  PDR  and  CDR  are  normally 
conducted  during  the  Full-Scale  Development  Phase  and  are  used  to  monitor 
the  developer's  design  progress  as  they  design  and  develop  the  associated 
CIs.  Primary  responsibility  for  the  conduct  of  design  reviews  is  assigned 
to  the  contractor  with  participation  from  the  PO. 

1.4  PURPOSE  OF  CONFIGURATION  MANAGEMENT  AUDITS 

Configuration  management  audits  are  used  to  determine  if  specific  contractual 
requirements  have  been  accomplished.  For  the  acquisition  of  CPCIs  all 
configuration  management  audits  are  associated  with  the  Full-Scale  Develop- 
ment contract.  The  primary  responsibility  for  the  conduct  of  configuration 
management  audits  belongs  to  the  PO  with  the  developer  playing  a support 
role. 

1 . 5 SEQUENCE  OF  SYSTEM/CPCI  REVIEWS  AND  AUDITS 

Figure  1 relates  system  and  CPCI-level  reviews  and  audits.  The  tasks  in  the 
Validation  and  Full-Scale  Development  Phases  can  be  accomplished  by  either 
industry  or  in-house  agencies.  The  term  "developer"  is  used  to  refer  to 
the  organization  responsible  for  performing  the  tasks,  whether  it  be  in-house 
or  contractor. 
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Figure  1.  System/CPCI  Reviews  and  Audits 


During  the  Validation  Phase  the  System  Specification  is  the  contract  specifica- 
tion governing  the  developer's  tasks.  As  the  developer  proceeds  from  the 
System  Specification  to  detailing  the  requirements  for  each  CPCI  in  a CPCI 
Development  (Part  I)  Specification,  the  PO  uses  the  SRR  and  SDR  to  monitor 
progress  between  the  baselines  (see  Figure  1).  The  basic  differences 
between  the  SRR  and  SDR  is  the  level  of  technical  detail  available  for  review. 
The  contract  specification  for  Full-Scale  Development  is  the  CPCI  Develop- 
ment (Part  I)  Specification  and  as  the  developer  designs  and  develops  the 
CPCI  the  PDR  and  CDR  are  used  to  monitor  his  progress.  Again  the  difference 
between  these  two  reviews  is  the  level  of  detail  available  for  review. 
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Subsequent  to  CDR  the  CPCI  is  coded  and  tested.  This  is  followed  by  an  FCA  to 
determine  if  the  CPCI  has  satisfied  the  requirements  of  the  CPCI  Development 
Specification  (i.e.,  is  a qualified  CPCI).  The  TCA  is  followed  by  a PCA  to 
determine  if  the  support  documentation  (e.g.,  users'  manual)  and  the  config- 
uration status  information  is  compatible  with  the  CPCI. 

If  the  CPCI  was  not  fully  qualified  prior  to  PCA  and  certain  requirements  must 
be  demonstrated  after  PCA,  then  this  is  accomplished  at  an  FQR.  When  a CPCI 
is  qualified  prior  to  PCA,  then  FQR  is  conducted  in  conjunction  with  FCA  (see 
Figure  1). 

The  SRR  and  SDR  are  reviews  conducted  at  the  system/segment  level;  all  other 
reviews  and  audits  arc  conducted  at  the  CPCI  level.  MIL-STD-1521A  (USAF) 
is  the  requirements  document  for  stating  contractual  requirements  for  reviews 
and  audits.  When  applying  these  requirements  to  a specific  contract  they 
must  be  tailored  to  the  particular  application  and  system  needs.  This  tailor- 
ing of  the  requirements  must  be  reflected  in  the  contract  statement  of  work 
(SOW). 


1.6  CONTENTS 

The  subsequent  contents  of  this  guidebook  consist  of  five  sections  and  two 
appendixes  as  follows: 

• Chapter  2 - General  Requirements.  Provides  guidance  to  the 
Software  Director  (SD)  for  preparing,  implementing,  and  conducting 
engineering  design  reviews  and  configuration  management  audits. 

It  identifies  the  responsibilities  of  the  participating  organiza- 
tions and  the  requirements  that  result  from  both  reviews  and 
audits. 

• Chapter  3 - Engineering  Design  Reviews.  Provides  detailed 
guidance  regarding  the  conduct  of  engineering  design  reviews. 

It  describes  the  purpose  of  each  review,  technical  information 
and  documentation  to  be  evaluated,  specific  requirements  for 
each  review,  and  post  review  activities. 

0 Chapter  4 - Configuration  Management  Audits.  Provides  detailed 
guidance  on  conducting  configuration  management  audits.  It 
describes  the  purpose  of  each  audit;  identifies  information, 
documentation,  and  products  to  be  audited;  and  identifies  post 
audit  actions.  ' 
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• Chapter  5 - FCA/FQR  and  PCA  Sample  Forms.  Contains  unofficial 
sample  forms,  taken  for  the  most  part  from  MIL-STD-1521A  (USAF), 
that  may  be  used  to  facilitate  the  collection  of  data  at  design 
reviews  and  audits. 

• Chapter  6 - Reviews  and  Audits  Problem  Areas.  Identifies  some 
of  the  common  and  more  serious  problems  associated  with  imple- 
menting and  conducting  engineering  design  and  reviews  and 
configuration  management  audits. 

• Appendix  A - Glossary.  Defines  terms  and  acronyms  used  in 
this  guidebook. 

• Appendix  B - Bibliography.  Provides  list  of  regulations, 
specifications,  and  standards  (RSS)  that  are  particularly 
relevant  to  software  reviews  and  audits. 
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The  general  requirements  for  reviews  and  audits  are  covered  in  Section  4 
of  MIL-STD-1 521A(USAF) , This  guidebook  assumes,  as  does  MIL-STD-1 b21 A (USAF), 
that  ^n  industry  contract(s)  is  awarded  for  the  Validation  Phase  and  that  the 
SRR  and  SDR  are  conducted  to  monitor  the  contractor's  progress. 

During  the  Validation  Phase,  engineering  design  reviews  are  conducted  on 
the  system/ system  segments  to  evaluate  the  Cl  definitions  and  to  determine  if 
system  design  compatibility  has  been  maintained.  During  the  Full-Scale 
Development  Phase,  engineering  design  reviews  and  configuration  management 
audits  are  conducted  on  individual  CIs  including  CPCIs. 

Subsequent  discussion  in  this  section  provides  interpretation  and  guidance 
regarding  general  requirements  as  defined  in  MIL-STD-1 521 A (USAF). 

2 . 1 LOCATION  AND  SCHEDULING 

According  to  Paragraph  4.1.2  of  MIL-STD-1 521A  (USAF),  design  reviews  and 
audits  are  generally  conducted  at  the  contractor's  facility.  While  this  is 
normally  true  for  hardware  CIs,  there  may  be  exceptions  for  CPCIs.  Support 
CPCIs,  such  as  compilers,  can  usually  be  qualified  and  accepted  at  the 
development  facility.  For  CPCIs  directly  associated  with  the  operational 
mission  (application  programs),  the  CPCI  will  normally  require  operationally 
configured  equipment  for  qualification  purposes.  The  FQT  and  FCA  are  there- 
fore normally  conducted  at  the  System  Development  Test  and  Evaluation  (DT&E) 
site.  PCAs  should  be  accomplished  at  the  development  facility  after  successful 
completion  of  the  FQT/FCA.  When  FQRs  are  conducted  after  PCA,  they  may  be 
scheduled  at  the  most  convenient  location,  e.g.,  the  System  DT&E  site. 

Otherwise  the  FQR  should  be  scheduled  in  conjunction  with  FCA.  The  principal 
criterion  for  review-site  selection  should  be  assurance  of  the  availability  of 
the  required  information  with  minimum  disruption  of  the  development  effort. 

2.2  RESPONSIBILITIES  OF  PARTICIPATING  ORGANIZATIONS 

Paragraphs  4.1.3  and  4.2  of  MIL-STD-1 521 A (USAF)  identify  both  contractor  and 
procuring  activity  requirements  for  supporting  reviews  and  audits.  It  is 
important  to  note  the  difference  in  emphasis  between  reviews  and  audits. 

Reviews  are  primarily  the  responsibility  of  the  developer  to  conduct  with  the 
PO  playing  a secondary  role  of  questioning  the  technical  information  being 
presented.  The  primary  responsibility  for  conducting  audits  is  assigned  to 
the  PO  with  the  developer  providing  the  necessary  support. 
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2 . 3 APPROVAL  RE5.U  1 RJMJNJS 

Approval  requirements  for  engineering  design  reviews  significantly  differ  from 
those  related  to  configuration  management  audits.  It  is  important  to  the  con- 
tractor to  successfully  complete  the  contract  milestones  because  his  progress 
payments  may  depend  upon  their  completion.  However,  the  PO  should  avoid  an 
indication  of  successful  completion  of  a review  or  audit  until  convinced  that 
the  contractual  requirements  for  a particular  milestone  have  been  satisfied. 

2.3.1  Engineer! ng  Desi gn  Revi ews 

The  purpose  of  a design  review  is  to  provide  the  PO  with  a way  to  monitor  the 
contractor's  technical  progress;  it  is  not  to  control  it.  The  PO's  responsi- 
bility is  to  participate  in  the  design  review  and  provide  feedback  to  the 
contractor  regarding  his  results.  The  PO  may  criticize  the  design,  coiiiiient 
on  its  ability  to  meet  performance  requirements,  or  ask  technical  questions 
about  the  design.  The  PO  should  not.  however,  offer  suggestions  or  guidance 
about  how  to  modify  the  design;  that  is  the  contractor's  domain  and  any  design 
suggestions,  even  oral  remarks,  may  be  construed  by  the  contractor  to  be 
guidance  or  direction  which  can  result  in  a "constructive  change"  to  the 
contract,  at  increased  cost  to  the  Air  Force  (see  6.1),  As  stated  in  Para- 
graph 4.2.4  of  MIL-STD-1521A  (USAF),  "the  procuring  activity  establishes  the 
adequacy  of  the  contractor's  review  performance  by  notification  of: 

• Approval--to  indicate  that  the  review  was  satisfactorily 
completed. 

• Contingent  Approval --to  indicate  that  the  review  is  not 
considered  accomplished  until  the  satisfactory  completion 
of  resultant  action  items. 

• Disapproval--to  indicate  that  the  review  was  seriously 
inadequate. " 

2.3.2  Con fj jju ration  Man agement  Aud i ts 

Configuration  management  audits  are  performed  to  confirm  that  contractual 
requirements  have  been  satisfied,  thus  requiring  Government  approval.  For 
example,  PCA  for  a CPCI  provides  for  acceptance  of  the  CPCI  and  upon  success- 
ful completion  the  transfer  of  ownership  from  developer  to  the  PO  takes  place. 
The  PO  will  notify  the  contractor  regarding  the  results  of  the  audit,  i.e., 
approval,  contingent  approval  or  disapproval.  If  the  contractor  is  worried 
that  his  progress  payments  nay  be  delayed  for  something  other  than  "approval," 
the  PO  will  normally  get  full  cooperation  in  correcting  any  deficiencies. 
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2.4  CONDUCTING  DESIGN  REVIEWS  AND  AUDITS 

Many  problems  are  created,  particularly  at  design  reviews,  if  attendees 
focus  only  on  the  technical  aspects  of  the  review  and  are  insensitive  to 
contractual  implications.  The  SD  must  recognize  that  he  is  not  the 
technical  manager  of  an  in-house  software  development  project  and  must 
therefore  be  extremely  sensitive  to  the  defined  and  agreed  upon  contractual 
relationships  and  responsibilities.  The  following  list  identifies  potential 
problems: 

• During  Full-Scale  Development,  the  contractor  is  required  to 
satisfy  the  requirements  of  the  CPCI  Development  (Part  I) 
Specification  by  creating  a design  that  will  satisfy  these 
requirements.  During  the  PDR  and  CDR,  the  SD's  responsibility 
is  to  determine  if  the  design  will  meet  the  performance  require- 
ments. If  the  contractor's  design  does  not  do  this,  then  the 
SD's  function  is  to  so  inform  the  contractor  and  direct  him  to 
correct  it.  However,  the  SD  should  not  dictate  a specific  design 
approach. 

• During  the  Validation  Phase,  when  contractors  are  involved  in  a 
competitive  situation,  any  information  given  to  one  contractor 
at  the  SRR  or  SDR  must  be  made  available  to  the  other  contrac- 
tors if  the  information  impacts  the  competion.  A contractor 
in  a competitive  situation  cannot  be  given  information  which 
may  give  him  an  advantage  over  his  competitors. 

fl  The  SD  must  be  sensitive  to  the  use  of  the  word  "approve"  in  the 
context  of  acquisition  management.  When  the  Government  gives  its 
approval  on  some  aspect  of  the  contract  it  is  basically  endorsing 
it.  This  approval  may  very  well  establish  a shift  in  responsi- 
bility from  contractor  to  Government,  e.g.,  when  the  Government 
approves  a contractor's  specification,  the  Government  becomes 
responsible  for  the  contents  and  the  contractor  becomes  respon- 
sible for  implementing  the  specification.  If  approval  is 
given  to  the  design  approach  presented  at  PDR  by  the  contractor, 
the  contractor's  responsibility  is  to  implement  that  approved 
design,  rather  than  satisfy  the  performance  requirements  of  the 
CPCI  Development  Specification.  This  is  why  it  is  policy  not 
to  approve  anything  at  design  reviews  other  than  the  minutes. 

The  approval  of  the  minutes  indicates  that  the  review  was 
satisfactorily  completed. 
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During  configuration  managenient  audits  the  PO  must  assume 
responsibility  for  conducting  the  audit  and  the  contractor 
provides  support.  Section  4 of  MIL-STD-1 521 A(USAF)  is 
somewhat  misleading  on  this  point  because  it  lumps  reviews 
and  audits  together.  During  the  audits  the  PO  verifies 
that  the  contractual  requirements  have  been  satisfied.  The 
FCA  and  PCA  are  the  prime  instruments  leading  to  "contractual 
acceptance"  of  the  deliverable  items.  When  the  Air  Force 
accepts  each  deliverable  from  the  contractor,  it  indicates 
that  the  contractor  has  fulfilled  his  contractual  obligations 
regarding  that  item;  acceptance  is  final . If  the  contractor 
is  allowed  to  take  the  lead  roTe~Tn  audits,  and  there  are 
deficiencies  in  the  items  to  be  delivered,  he  may  attempt  to 
cover  up  the  deficiencies.  Normally  the  Government  will 
attempt  to  protect  itself  by  having  a "latent  defects"  clause 
specified  in  the  contract.  Latent  defects  are  defects  which 
exist  at  the  time  of  acceptance  but  are  not  discoverable  by 
a reasonable  inspection.  Once  a latent  defect  has  been 
identified  it  is  the  contractor's  responsibility  to  correct 
the  defect.  In  practice,  it  has  proven  difficult  to  establish 
that  a latent  defect  did  exist  so  the  degree  of  protection 
offered  by  this  clause  is  moot.  CPCI  acceptance  is  normally 
executed  by  signing  the  Form  DD  250.  At  the  time  of  signing, 
specific  shortages  should  be  listed.  The  contractor  must 
correct  these  shortages  prior  to  FQR.  The  corrections  should 
be  noted  at  FQR. 


SECTION  3 - ENGINEERING  DESIGN  REVIEWS 


Design  reviews  fall  into  two  categories,  system-level  reviews  and  Cl-level 
reviews,  as  follows: 

• System-Level  Reviews  (SRR  and  SDR).  During  the  Validation  Phase 
the  System/ System'  Segment  Specification  is  the  contract 
baseline.  The  primary  activity  during  this  phase  is 
system  engineering;  specifically,  the  system  requirements  are 
being  refined  and  allocated  to  CIs.  The  purpose  of  the  system- 
level  reviews  is  to  monitor  the  contractor's  system  engineering 
activities  to  determine  if  system  integrity  is  being  maintained  and 
if  Cl  definition  is  consistent  and  compatible  with  the  system 
requirements . 

0 Cl -Level  Reviews  (PDR  and  CDR).  During  the  Full-Scale  Development 
^ase  the  Cl  Development  (Part  I)  Specification  is  the  contract 
specification.  The  Development  Specification  defines  the  Cl 
performance  requirements  and  interfaces.  In  the  case  of  CPCIs, 
the  primary  activity  during  this  phase  is  the  design  and  development 
of  computer  programs.  The  CPC  I is  designed  and  coded,  using  a 
CPCI  Development  Specification  as  the  design-to  requirement.  The 
purpose  of  the  Cl-level  reviews  is  to  determine  if  the  selected 
design  is  proceeding  in  a manner  which  is  compatible  with  the  CPCI 
Development  Specification. 

Each  of  these  reviews  is  discussed  in  the  following  paragraphs  in  terms  of 
(1)  materials  to  be  reviewed,  (2)  requirements,  and  (3)  post-review  activities. 

The  manner  in  which  the  system  level  reviews  are  conducted  will  depend  on  how 
the  Validation  Phase  is  being  performed.  If  the  Validation  Phase  is  being 
performed  by  contractors,  competitively,  care  must  be  exercised  by  PO  personnel 
not  to  jeopardize  the  competitive  situation.  MIL-STD-1 521A  (USAF)  does  not 
distinguish  between  the  competitive  and  non-competitive  situation.  It  is 
best  to  check  with  procurement  personnel  regarding  the  treatment  of  competitive 
information  for  each  program  prior  to  conducting  these  reviews. 

3 . 1 SYSTEM  REQUIREMENTS  REVIEW 

The  SRR  is  an  in-process  review,  i.e.,  it  can  be  conducted  several  times 
during  the  system  life  cycle.  Normally,  only  one  SRR  is  scheduled  at  the 
early  stages  of  the  Validation  Phase  activities.  Appendix  A of  MIL-STD-1 521 A 
(USAF)  assumes  a series  of  SRRs  and  describes  SRR  requirements.  These 
requirements  must  be  examined  and  tailored  to  the  needs  of  each  program.  If 
an  SRR  is  scheduled  at  the  early  stages  of  the  Validation  Phase  the  majority 
of  the  items  listed  in  Appendix  A will  not  be  available,  e.g.,  (1)  configura- 
tion management  plan,  (2)  data  management  plan,  (3)  engineering  integration. 
Figure  1,  Engineering  and  Test  Flow  in  MIL-STD-1 521 A (USAF),  shows  the  SRR 
being  conducted  during  the  Conceptual  Phase.  The  first  sentence  of  Paragraph 
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10.1  in  Appendix  A states  that  SRRs  are  normally  conducted  during  the 
system's  Conceptual  or  Validation  Phases.  However,  the  remainder  of 
Appendix  A states  requirements  that  can  only  be  satisfied  after  the 
completion  of  the  Conceptual  Phase.  Paragraph  10.2  of  Appendix  A states 
that,  "The  total  system  engineering  management  activity  and  its  output 
shall  be  reviewed  for  responsiveness  to  the  statement  of  work  and  system  or 
system  segment  requirements".  This  can  only  be  satisfied  during  the 
Validation  Phase,  and  therefore  SRRs  should  be  scheduled  starting  in  the 
Validation  Phase. 

The  purpose  of  an  SRR  is  to  evaluate  the  progress  and  direction  of  the 
initial  system  engineering  effort  during  the  Validation  Phase  rather  than 
to  conduct  an  on-the-spot  technical  integration  of  the  system.  This  review 
provides  the  PO  with  visibility  regarding  contractor  performance  within  the 
scope  of  the  Validation  Phase  contract. 

Chapter  8,  Paragraph  8-13,  of  AFSCP  800-3  states  that  "SRRs  may  result  in 
technical  or  engineering  management  realignment  to  ensure  that  the  contrac- 
tor's initial  technical  interpretation  of  the  contract  is  in  line  with  pro- 
gram objectives."  When  applying  this  statement  in  a competitive  situation, 
the  PO  must  ensure  that  any  realignment  does  not  interfere  with  the  competi- 
tion. Whet,  in  doubt,  seek  legal  advice  before  proceeding. 


3.1.1  Materials  to  be  Reviewed 

Since  the  SRR  is  normally  conducted  during  the  early  stages  of  the 
Validation  Phase,  the  PO  should  not  expect  complete  formal  documentation. 

The  review  should  be  conducted  using  preliminary  documentation.  These 
documents  will  normally  be  in  the  contractor's  working  paper  format.  If 
the  SRR  is  delayed  until  formal  documentation  is  available  then  the  direction 
of  the  Validation  Phase  approach  may  be  fixed  and  it  may  be  too  late  to 
influence  it.  Appendix  A of  MIL-STD-1521A  (USAF)  provides  a relatively 
exhaustive  list  of  information  to  be  reviewed  at  SRRs.  However,  it  assumes 
that  the  requirement?  deal  with  a series  of  SRRs.  For  the  initial  SRR  only 
a subset  of  this  data  may  be  available  depending  on  when  it  is  scheduled. 

For  the  initial  SRR  the  following  subset  of  contractor-produced  data  is 
normally  reviewed: 

t Mission  and  requirements  analysis 

• System-level  functional  flow  analysis 

• Preliminary  requirements  allocation  to  CIs 

• System  interface  studies 

• Initial  trade  studies 
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3.1.2  Requirements 


The  contracts  awarded  for  the  Validation  Phase  initiate  contractor  technical 
efforts  to  expand  the  total  requirements  of  the  system  to  the  CPCI  level. 

From  this  point,  the  system  acquisition  process  proceeds  on  contractual 
schedules  toward  definition  of  the  Allocated  Baseline  and  associated  plans 
which  will  scope  and  pace  the  total  technical  effort  for  the  Full-Scale 
Development  Phase.  The  basic  purpose  of  conducting  the  SRR  during  the  early 
stages  of  the  Validation  Phase  is  to  determine  if  contractor  efforts  satisfy 
the  tasks  prescribed  by  the  SOW,  and  to  determine  if  the  interpretations  and 
detailing  of  the  system  requirements  are  compatible  and  consistent  with  the 
System  Specification,  rather  than  exceed  or  fall  short  of  the  stated  require- 
ments. This  entails: 

• Reviewing  the  initial  trade  studies  performed  by  the  contractor 
to  see  if  they  support  the  allocation  of  functional  require- 
ments to  the  CPC Is. 

• Reviewing  preliminary  CPCI  selection  to  assure  the  optimum 
selection  for  future  development. 


Trade  Studies 


To  arrive  at  a preliminary  selection  of  CIs  and  CPCIs,  the  contractor 
normally  conducts  initial  trade  studies.  When  contracting  to  a segment 
System  Specification,  the  trade  studies  are  confined  to  the  boundaries  of 
the  segment  on  which  the  contractor  is  working.  A system  segment  is  a 
discrete  package  of  system  performance  requirements,  functional  interfaces, 
and,  eventually,  CIs  contracted  to  one  contractor  or  assigned  to  one  Govern- 
ment organization  which  is  directly  responsible  to  the  procuring  agency  for 
that  part  of  a system's  performance.  The  reason  for  the  statement  "and, 
eventually  CIs"  is  to  indicate  that  at  this  point  within  the  system 
acquisition  life  cycle  the  CIs  have  not  yet  been  defined.  The  segment 
definitions  are  based  on  system  engineering  decisions  made  during  the 
Conceptual  Phase  when  the  system  requirements  were  apportioned  into  segment 
packages.  The  extent  of  the  trade  studies  is  dependent  on  the  segment 
definitions.  For  example: 

t If  the  segment  contains  both  hardware  and  software,  then  the 
analysis  and  trade  studies  normally  involve  hardware,  software, 
man/machine,  and  manual  functions.  The  goal  is  to  support  the 
allocation  of  functions  to  these  four  areas  and  eventually  to 
define  the  CIs  for  hardware  and  software. 
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• When  the  software  is  contained  within  a segment  that  does  not  ! 

include  hardware,  then  the  analysis  and  trade  studies  are 

restricted  to  software,  man/machine,  and  manual  functions. 

Preliminary  Cl  Selection  i 

Although  at  SRR  the  CPCI  selections  are  preliminary,  the  PO  should  pay  \ 

particular  attention  to  these  selections  and  evaluate  them  in  terms  of  1 

their  impact  on  future  activities.  Section  2 of  the  Configuration  Management  | 

guidebook  provides  the  rationale  used  in  selecting  CPCIs.  Although  the 
Initial  CPCI  breakout  is  done  on  a technical  basis,  the  final  decision  must 
be  tempered  by  consideration  of  management  and  acquisition  considerations. 

The  PO  should  review  the  CPCI  selections  to  determine: 

• If  they  are  at  an  appropriate  level  for  management  control 
purposes.  The  CPCI  is  the  basis  for  management  reporting 
against  the  WBS. 

t Their  impact  on  configuration  management.  How  does  the  number 
of  CPCIs  affect  the  number  of  baselines,  change  processing 
requirements,  configuration  management  status  accounting,  and 
reporting?  Normally  these  are  organized  by  CPCI  and  the  initial 
definitions  should  be  examined  to  determine  their  effect  on  the 
costs  of  conducting  configuration  management. 

• Their  impact  on  data  management,  i.e.,  data  selection  and  data 
costs.  Each  CPCI  has  its  own  support  data  packages.  For 
example,  for  each  CPCI,  there  is  a Development  Specification; 

Product  Specification;  test  plan,  procedures,  and  reports;  users 
manuals;  and  version  description  documents. 

• Their  test  management  implications.  The  early  phase  of  DT&E 
are  directed  to  CPCI-level  testing  and  are  the  responsibility 
of  the  developer.  CPCI  selection  has  an  impact  on  test  planning 
activities  and  test  responsibilities. 

0 Their  impact  on  engineering  management.  The  definition  of  a 
CPCI  identifies  an  integrated  design  package  to  he  designed, 
reviewed,  and  developed  during  the  Full-Scale  Development 
contract . 


^ I 


18 


I 


I 

• Their  impact  on  CPCI  delivery  and  integration.  A CPCI  identifies 
the  level  of  delivery  for  the  product,  i.e.,  acceptance  is 
accomplished  at  the  CPCI  level.  The  CPCI  definition  defines  the 
requirements  for  each  design  package  that  will  be  delivered  and 
integrated  with  the  other  CIs.  Naturally  this  definition 
establishes  development  and  integration  responsibilities  that 
will  eventually  be  reflected  in  the  Full-Scale  Development 
contract . 

Many  acquisition  problems  are  created  if  CPCI  definitions  are  only  viewed 
technically.  A common  misconception  is  that  more  CIs  will  provide  more 
control  and  visibility.  In  fact,  when  many  CPCIs  are  defined,  costs  are 
increased  (increased  data,  management  reviews,  and  control  and  configuration 
management  requirements),  the  PO  accepts  more  responsibility  (an  increase 
in  the  number  of  Development  Specifications,  each  with  interfaces  that  must 
be  approved  by  the  Government  when  the  specification  is  baselined),  and  the 
visibility  remains  essentially  the  same  (the  Development  Specification 
requirements  must  still  be  defined  at  the  performance  level).  Multiple 
CPCIs  result  in: 

• Instead  of  one  CPCI  Development  Specification,  the  CPCI 
requirements  are  contained  in  several.  The  level  of  control, 
when  baseTined,  is  still  performance  requirements. 

• Each  Development  Specification  has  added  interfaces,  i.e., 
interfaces  between  CPCIs.  When  the  Allocated  Baseline  for  each 
CPCI  is  established,  the  PO  accepts  responsibility  for  these 
interfaces.  The  contractor's  responsibility  is  to  implement 
the  requirements  for  each  specification. 

0 The  configuration  management  activity  now  has  to  control  more 
baselines  and  more  sets  of  configuration  control  and  reporting 
records . 

0 The  support  data  package  will  contain  a substantial  increase 
in  the  number  of  documents,  i.e.,  each  CPCI  will  have  a 
test  plan,  test  procedures,  and  test  reports,  in  addition  to 
a CPCI  Product  Specification. 

0 At  the  PDRs,  the  design  approach  for  each  CPCI  is  presented. 

Whether  the  design  of  all  the  CPCIs  is  compatible  is  now  the 
PC's  concern. 

0 At  the  time  of  delivery  the  contractor  delivers  multiple  CPCIs 
with  data  packages.  The  PO  now  has  to  arrange  to  have  them 
integrated,  which  the  developer  may  be  contracted  to  perform 
after  he  satisfies  his  responsibility  to  the  CPCI  Development 
Specifications. 


3.1.3 


Post-Review  Activities 


At  the  completion  of  the  SRR,  the  contractor  will  draft  the  minutes  and  have 
them  reviewed  and  approved  by  the  co-chairmen.  The  minutes  are  prepared  in 
accordance  with  DI-E-3118.  Normally  this  DID  is  a contract  data  requirement 
specifying  that  the  minutes  are  to  be  supplied  by  the  contractor.  Approval  of 
the  minutes  indicates  that  they  accurately  reflect  the  results  of  the  meetinn. 
The  last  section  of  the  minutes,  "Problem  Study  Areas,"  provides  an  identifi- 
cation of  each  problem/deficiency  found.  Each  problem  identified  for  post- 
review action  should  be  assigned  an  action  item  number,  a responsible  organ- 
zation,  and  a suspense  date.  Each  problem  should  be  tracked,  and  proposed 
solutions  reviewed  by  the  PO.  If  the  PO  deems  the  solution  inadequate, 
the  action  item  should  remain  open  until  a satisfactory  solution  is  developed. 

3.2  SYSTEM  DESIGN  REVIEW 


The  SDR  is  conducted  during  the  Validation  Phase  before  the  developer  submits 
his  Validation  Phase  products.  The  purpose  of  the  SDR  is  to  allow  the  PO  to 
evaluate  the  adequacy  of  these  products  prior  to  submission.  For  systems 
with  no  scheduled  Validation  Phase,  an  SDR  should  be  scheduled  during  the 
early  stages  of  the  Full-Scale  Development  Phase.  Even  when  there  is  no 
Validation  Phase,  the  CPCI  Development  Specifications  are  still  required,  and 
the  activities  normally  conducted  during  the  Validation  Phase  are  scheduled 
during  the  early  stages  of  the  Full-Scale  Development  Phase.  Appendix  B 
of  MIL-STD-1 521 A (USAF)  identifies  requirements  for  the  SDR. 

3.2.1  Materials  to  be  Reviewed 

The  SDR  for  the  software  portions  of  system  segments  should  include  a review 
of  the  following  materials: 

• Preliminary  Development  Specification  for  each  CPCI  (DI-E-3119A) 

• Updating  information  for  the  System/Segment  Specification 
(DI-E-3101) 

• Trade  studies  (DI-S-3606) 

• Quantitative  and  Qualitative  Personnel  Requirements 
Information  - QQPRI  (DI-H-3253) 

§ Computer  Program  Development  Plan-CPDP  [DI-S-30567] 

• Test  planning  information  (DI-T-3703) 

• Operator  procedures  and  task  analysis  report  (DI-H-3268A) 

• Exercise  capability  implementation  information  (DI-H-3270A) 

• Recommendations  for  equipment,  facilities,  and  communications 
(usually  documented  as  an  integral  part  of  the  technical  proposal) 
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3.2.2  Requirements 


The  basic  requirement  of  the  SDR  is  to  evaluate  the  contractor's  progress 
just  prior  to  the  formal  submission  of  Validation  Phase  products.  It  is 
essential  that  the  documentation  reviewed  at  SDR  be  critically  examined  to 
assure  attainment  of  the  objectives  of  the  Validation  Phase.  The  contract 
specification  during  the  Validation  Phase  should  be  either  the  System  or  a 
Segment  Specification.  The  developer's  task  is  to  detail  the  requirements 
and  allocate  them  to  CPCIs.  For  each  CPCI  a Development  Specification  will 
be  generated  along  with  a proposal  for  conducting  a Full-Scale  Development 
Phase  contract.  The  PO  will  participate  in  the  SDR,  normally  supported  by 
the  General  System  Engineering/Technical  Director  (GSE/TD)  contractor*, 
the  PO's  objectives  are  to  determine  (1)  that  the  requirements  for  each 
system  element  are  essential  and  reflect  the  requirements  of  the  System 
Specification,  (2)  that  the  system  definition  presents  an  optimum  balance  of 
system  elements,  and  (3)  that  a technical  understanding  has  been  reached. 
Specifically,  the  review  should  include  evaluation  of  the  following  data: 

• Trade  Studies.  The  contractor  shall  accomplish  the  trade  studies 
specifically  directed  by  the  Validation  Phase  contract  which 
are  to  provide  technical  substantiation  of  requirements  for 
CPCIs,  equipment,  personnel,  and  training.  The  trade  studies 
shall  consider  the  availability  of  current  computer  programs, 
computing  equipment  (either  within  the  Government  inventory  or 
commercially  available),  personnel  with  necessary  skills,  and 
existing  training  programs.  The  objective  of  these  trade  studies 
is  to  achieve  a system  balance  based  on  the  consideration  of 
total  requirements.  The  results  will  be  documented  in  accord- 
ance with  DI-S-3606.  There  is  no  standard  set  of  criteria  for 
evaluating  trade  studies  as  the  criteria  depend  on  the  needs 
of  each  specific  system.  The  trade  study  conclusions  should 
identify  the  rationale  for: 

- Allocation  of  functions  tn  CPCIs. 

- Assignment  of  manual  functions. 

- Allocation  of  requirements  to  hardware. 

- Man/machine  decisions,  including  display  format 
and  contents,  push-button  allocation  and  assign- 
ment, and  report  formats  and  contents. 

- Alternate  information  flow. 

- Sizing  and  timing  conclusions. 


*The  mission  of  the  GSE/TD  contractor  is  defined  in  AFSCP  800-3, 
paragraph  20-11,  page  20-10. 
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For  each  trade  study,  the  PO  should  assure  that: 

- Trade  study  objectives  are  identified. 

- Justification  for  parameters  and  weighting  factors 
is  presented. 

- All  requirements  and  constraints  are  identified. 

- All  results  were  logically  derived. 

- Total  system  implications  were  considered. 

- Alternative  approaches  were  objectively  considered 
and  evaluated. 

- The  selected  set  of  requirements  meets  rather  than 
exceeds  requirements. 

- Any  requirements  imposed  on  the  system  by  the  trade 
studies  are  properly  communicated. 


• System  Engineering  Documentation. 

- Functional  allocation,  including  computer  program  functions, 
operator  functions,  and  man/machine  functions. 

- Operator  task  analysis. 

- Training  and  evaluation  needs  analysis. 

' System  exercising  capability  implementation  plan. 

- Built-in  simulation  and  test  capabilities. 


- Data  reduction  capabilities. 

- Computer  program  development  tools. 

- Computer  Program  Development  Plan. 

- Personnel  requirements,  including  operational  requirements, 
computer  program  support  requirements,  and  simulation/ 
exercising  personnel  requirements. 


- Recommended  design  requirements  for  equipment,  communications, 
and  faci 1 ities . 

- CPCI  Development  (Part  I)  Specifications,  including 
operational  requirements,  support  requirements,  and 
utility  requirements. 

- System  Specification  inputs,  including  a list  of  CPCIs  and 
Functional  allocations. 
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In  evaluating  the  system  engineering  documentation,  the  PO  should 

assure  that: 

- The  CPCI  selection  was  accomplished  using  the  rationale  and 
criteria  defined  in  Section  2 of  the  Configuration  Manage- 
ment guidebook. 

- The  operator  task  analysis  results  are  compatible  with  the 
operator  interface  requirements  of  the  CPCI  Development 
Specifications,  i.e.,  keyboard  actions  and  display  require- 
ments . 

- The  duty  positions  and  associated  operator  qualifications 
are  identified. 

- The  training  requirements  have  been  identified  for  operational 
and  support  personnel . 

- An  adequate  test  program  has  been  established  for  the 
CPCI(s). 

The  system  exercising  capability  has  been  reviewed  to  identify 

the  following: 

- System  elements{s)  to  be  exercised. 

- System  personnel  to  be  exercised. 

- Required  exercising  capability  elements(s). 

- Required  exercising  personnel  . 

- The  training/evaluation  needs  to  be  satisfied. 

In  evaluating  each  CPCI  Development  Specification,  the  PO  should 

assure  that: 

- The  Development  Specification(s)  contain  performance  require- 
ments rather  than  computer  program  design  information,  i.e., 
a non -programmer  can  evaluate  the  Development. Specification. 
Personnel  with  air  defense  knowledge  should  be  able  to 
evaluate  a Development  Specification  for  an  air  defense 
application  CPCI  since  it. describes  air  defense  requirements, 
not  computer  program  design  requirements. 

- The  Operational  CPCI  Development  Specification  reflects  an 
understanding  of  the  operational  mission  (see  Requirements 
Specification  guidebook). 

- The  requirements  are  defined  at  a level  of  detail  sufficient 
to  initiate  the  CPCI  design  effort. 
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The  CPCI  functional  flow  charts  integrate  the  CPCI  functions 
and  define  the  relationship  of  the  functions  to  the  CPCI 
external  interfaces. 

Interface  block  diagrams  identify  the  interfaces  between  the 
CPCI  and  its  immediate  operating  environment. 

The  CPCI  requirements  are  compatible  and  traceable  to  the 
System  Specifications. 

The  CPCI  interfaces  are  identified  and  detailed  in  terms 
of  specific  message  format,  contents,  and  timing  require- 
ments . 

The  explanations  for  "not  applicable"  or  "to  be  determined" 
entries  in  the  Development  Specifications  are  justified. 

Only  necessary  design  constraints  are  specified. 

The  display  requirements  are  defined  in  terms  of  display 
formats,  contents,  timing,  and  priorities. 

The  requirements  for  numbers,  positions,  labels,  and 
functions  of  operator  input  devices  are  specified,  e.g., 
push-button  layout  of  operator  console. 

The  following  information  is  available  relative  to  displays: 

a.  Formats  tc*be  us^d  in  presenting  information 

b.  Conventions  and  symbols  to  be  employed  in  presenting 
information  details 

c.  Rules  for  routing  displays 

d.  Rules  for  forcing  displays 

e.  Rules  for  updating  displays 

f.  Rules  for  priority  handling  of  displays 

Manual  input  device  infornation  is  identified,  including: 

a.  Allocation  of  the  types  of  information  to  be  inserted 
at  each  manual  input  device. 

b.  Frequencies  of  operator  insertion  readout  by  computer 
program. 

c.  Specific  details  of  each  action  to  be  available  for 
keyboard  type  devices. 


d.  Rules  for  feedback  to  inform  operating  personnel 
of  the  adequacy  of  their  insertions. 

- The  data  base  requirements  are  specified  in  terms  of  units 
of  measure,  range  of  possible  values,  precision,  and 
accuracy  of  requirements. 

- All  requirements  in  Section  3 of  the  Developmert  Specification 
are  accounted  for  in  Section  4. 

- All  applicable  documents  listed  in  Section  2 are  referenced 
in  Section  3 requirements. 

This  list  should  be  used  to  augment  the  requirements  in  Appendix  B 

of  MIL-STD-1521A  (USAF). 

In  reviewing  the  contractor's  proposed  updates  of  the  System 

Specification,  the  PO  should  assure  that: 

- The  updates  are  compatible  with  the  segment  requirements. 

- There  is  traceability  between  the  segment  requirements 
and  the  defined  CPCIs. 

- The  functional  interfaces  with  the  other  segments 
have  been  detailed. 

- The  expansion  has  provided  additional  detail  to  clarify 
and  amplify  basic  requirements,  rather  than  change  them. 

- Any  changes  to  the  system  requirements  have  been  identified. 

3.2.3  Post-Review  Activities 

Any  deficiencies  identified  during  the  SDR  should  be  noted  in  the  SDR 
minutes.  The  contractor  should  correct  the  deficiencies  prior  to  submitting 
the  Validation  Phase  products.  The  PO  should  review  the  SDR  minutes,  during 
final  evaluation  of  the  submitted  products,  to  identify  outstanding  problems 
and  to  determine  if  they  have  been  solved  in  the  delivered  products. 

3.3  PRELIMINARY  DESIGN  REVIEW 

The  PDR  is  a formal  design  review  conducted  during  the  Full-Scale  Development 
Phase.  Prior  to  PDR,  the  CPCI  Development  Specification  should  have  been 
authenticated  and  baselined.  The  PDR  is  a review  of  the  developer's  top- 
level  CPCI  design  in  response  to  the  approved  performance  requirements 
(CPCI  Development  Specification).  The  purpose  of  the  PDR  is  to  determine  if 


(1)  the  design  approach  takes  Into  account  all  performance  requirements,  (2) 
the  design  approach  will  satisfy  the  performance  requirements,  and  (3)  the 
CPCI  being  reviewed  is  functionally  compatible  (interfaces)  with  the  other  Cls. 
The  requirements  for  conducting  a PDR  are  specified  in  Appendix  C of  MIL-STD- 
1521A  (USAF).  These  requirements,  when  placed  on  contract,  should  be  tailored 
to  the  specific  needs  of  the  system/CPCl. 

3.3.1  Materials  to  be_R(jyiewed 

The  specific  technical  information  to  be  reviewed  and  the  details  concerning 
actions  to  be  accomplished  at  a given  PDR  are  normally  determined  by  the 
contractor  and  the  PO  when  the  agenda  for  the  review  is  established.  The 
material  to  be  reviewed  normally  includes: 

• Updated  CPCI  Development  Specification(s) 

• Storage  allocation  charts 

• CPCI  functional  flows 

• Functional  design  description  of  executive  control 
and  start/recovery  elements 

• CPCI  design  and  control  structure 

• Structure  and  organization  of  the  data  base 
t Interface  requiremc  ts 

0 Design  trade  studies 
0 ECP  status 

0 Computer  Program  Development  Plan 
0 CPCI  test  plan 

0 Development  support  requirements 
0 Development  tools 

3.3.2  Req^u  i remen  ts 

A PDR  is  normally  held  within  60  to  90  days  after  the  start  of  the  Full-Scale 
Development  Phase.  Only  one  PDR  is  required  to  be  successfully  accomplished 
for  each  Cl.  For  internal  management  purposes,  contractors  may  elect  to 
perform  additional  design  reviews  as  they  deem  necessary;  however,  formal 
notification,  participation,  and  acknowledgment  by  the  PO  is  not  required. 

The  PDR  shall  be  accomplished  when  the  basic  design  approach  has  been  selected 
by  the  contractor.  Normally  PDRs  are  accomplished  at  the  contractor's  facility 
where  the  design  is  in  progress.  The  design  approach  presented  at  the  PDR 
should  be  based  on  an  approved  sot  of  performance  requirements  contained  in 
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the  baselined  CPCI  Development  (Part  I)  Specification.  It  is  extremely 
important  that  the  Development  Specification  be  established  as  the 
Allocated  Baseline  prior  to  initiating  the  CPCI  design  approach.  The  require- 
ments in  MIL-STD-1 521A  (USAF)  assume  that  the  system  life  cycle  includes  a 
Validation  Phase.  Systems  not  having  a Validation  Phase  require  a significant 
tailoring  of  the  requirements.  The  basic  tasks,  normally  accomplished  within 
a Validation  Phase,  are  not  scheduled  in  the  early  stages  of  Full-Scale 
Development.  When  faced  with  this  situation  it  is  important  that  the  CPCI 
Development  Specification  be  approved  and  baselined  prior  to  PDR.  Some  system 
acquisitions  have  experienced  problems  when  the  approval  of  the  Development 
Specification  was  accomplished  during  PDR.  When  this  happens  the  developer 
has  to  create  a design  based  on  unapproved  requirements.  This  results  in  the 
developer  presenting  both  the  Development  Specification  and  a design  approach 
at  PDR.  If  the  requirements  within  the  specification  are  altered  as  a result 
of  evaluation  and  approval  this  may  have  an  impact  on  the  design  presented  at 
PDR.  Depending  on  the  magnitude  of  the  changes  there  may  be  a severe  impact 
on  schedules. 

The  PDR  for  CPCIs  is  accomplished  by  reviewing  the  following: 

0 Development  Specifications.  The  version  of  the  Development 
Specification  basel ined  at  the  end  of  the  Validation  Phase 
normally  will  contain  several  "To  Be  Determined"  (TBD)  state- 
ments. During  a competitive  Validation  Phase,  many  interfaces 
are  not  known  until  after  source  selection  at  the  end  of  the 
phase.  All  TBDs  should  be  removed  prior  to  the  end  of  PDR. 

0 Storage  Allocation  Charts.  These  charts  describe  the  manner  in 
which  various  elements  of  the  CPCI  have  been  allocated  to 
available  computer  storage.  In  cases  where  the  system  does 
not  have  fixed  storage  a review  of  the  algorithims  used  to 
determine  storage  should  be  accomplished.  The  intent  of  this 
part  of  the  review  is  to  ensure  the  compatibility  between 
the  CPCI  and  available  computer  storage. 

0 CPCI  Functional  Flows.  The  CPCI  ■'functional  flow  summarizes  the 
flow  of  information  for  the  complete  CPCI  and  identifies  the 
Computer  Program  Components  (CPCs),  their  relationship  within 
the  CPCI,  and  with  the  CPCI's  external  interfaces.  These 
charts  provide  an  overview  of  tasks  to  be  performed  by  the 
CPCI  and  can  be  related  to  the  performance  requirements 
in  the  Development  Specification. 
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• structure  and  Organization  of  Data  Base.  This  is  a description 
of  the  structural  layout  and  the  allocation  of  storage  require- 
ments of  the  CPCI  data  base,  identifying  the  types  and  charac- 
teristics of  data. 

• Interface  Requirements . Although  these  were  identified  in  the 
CPCI  Development  Specification  and  approved,  it  is  important 
that  they  be  reviewed  again  at  this  time.  Interface  requirements 
include  such  things  as  definitions  of  word  length,  message 
formats,  available  computer  storage,  and  timing  considerations. 

• Design  Trade  Studies.  This  information  was  developed  as  a 
result  of  studying  alternate  design  methods  and  techniques. 

• ECP  Status.  Review  the  configuration  management  accounting 
records  to  determine  which  approved  ECPs  should  be  reflected  in 
the  design  presented  at  PDR  and  to  ensure  that  they  are  included. 

0 CPDP.  The  CPDP  was  originally  generated  during  the  Validation 
Phase.  A design  approach  should  have  been  presented  in  the 
CPDP.  The  design  approach  presented  at  PDR  may  differ  from  the 
one  described  in  the  CPDP.  The  CPDP  should  be  reviewed  to 
determine  if  it  requires  updating  to  make  it  compatible  with 
current  design  and  development  approaches. 

0 Test  Plan.  The  initial  test  plan  was  submitted  as  one  of  the 
Validation  Phase  products.  At  PDR  it  should  be  updated  to 
reflect  any  changes  resulting  from  the  design. 

0 Support  Tools.  Requirements  for  tools  to  support  the  CPCI 
development  should  be  reviewed  for  their  adequacy  and  to 
determine  which  ones  should  be  required  to  support  the  CPCI 
during  deployment. 

The  PO  should  assure  that: 

0 TBDs  in  the  CPCI  Development  Specification  are  removed  and 
that  the  required  information  is  now  available  and 
adequate. 

0 There  is  compatibility  between  the  CPCI  and  the  other  CIs. 

0 There  is  traceability  between  the  requirements  in  the 
development  specification  and  the  design  approach. 

0 All  ECPs  approved  since  establishing  the  allocated  baseline 
are  reflected  in  the  PDR  design  information. 
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• The  planned  sequence  of  operations  for  the  CPCs  is 
compatible  with  the  requirements  of  the  Development 
Specification. 

• The  CPCI  timing  and  storage  requirements  are  consistent 
with  the  system  and  equipment  constraints. 

• The  design  constraints,  stated  in  the  Development  Specification 
have  been  incorporated  into  the  design  approach. 

• The  executive  control  design  satisfies  the  needs  of 
the  CPCI. 

t Design  considerations  have  been  made  to  accommodate 
adaptation  data  for  multi-site  systems. 

• Startover/recovery  features  meet  their  performance  requirements. 

• Design  trade  studies  have  been  accomplished  and  are 
reflected  in  the  selection  of  the  design  approach. 

• The  required  storage  has  been  allocated  to  the  CPCs  and 
the  data  base. 

• The  contractor  either  has  or  is  developing  any  required 
special  tools  needed  to  support  his  design  and  development 
effort. 

• There  is  a requirement  to  deliver/qualify  the  support  tools. 

• Any  differences  in  configuration  between  the  development 
facility  and  the  operational  equipment  configuration  are 
identified. 

• The  contractor  has  adequate  plans  to  accommodate  any  identified 
differences  in  configuration. 

3.3.3  Post-Review  Activities 

The  PO  has  several  decisions  that  may  be  made  as  a result  of  the  review, 
including: 

• Unqualified  approval  to  specify  complete  agreement 
between  and  among  the  reviewing  team  and  the  developer. 

• Approval  with  contingent  action  items  when  the  review 

is  not  considered  accomplished  until  satisfactory  completion 
of  actions. 
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0 Approval  with  deviations  when  it  is  in  the  interest  of  the 

program  to  award  limited  approval  and  protect  program  schedules 
pending  completion  of  further  engineering  as  indicated  by 
actions  assigned  in  the  minutes. 

0 Disapproval  when  the  subject  of  the  review  is  unsatisfactory 
or  generally  inadequate.  The  review  must  be  rescheduled  as 
a result  of  the  disapproval. 


The  contractor  will  prepare  and  submit  the  PDR  minutes,  normally  within  5 
working  days  after  completion  of  the  review.  The  PO's  approval  of  the  minutes 
indicates  that  the  minutes  accurately  record  the  actions  taken  at  the  review 
meeting.  The  PO  approval  of  the  minutes  does  not  indicate  approval  of  the 
design  presented.  When  the  minutes  identify  problem  areas  and  action  items, 
the  PO  should  assign  action  item  numbers  to  each  and  assign  personnel  to 
monitor  the  action  items. 

If  any  changes  to  the  established  baselines  are  required  as  a result  of  the 
PDR,  the  PO  should  assure  that  the  necessary  ECP  action  is  initiated.  Normally, 
some  change  requirements  are  identified  at  PDR  and  ECP  activities  should  be 
expected. 

The  Configuration  Management  Division  should  take  the  necessary  action  to 
update  the  Allocated  Baseline  to  reflect  the  updated  CPCI  Development  Speci- 
fication . 

3 . 4 CRITICAL  DESIGN  REVIEW 

Upon  completion  of  the  PDR  the  developer  initiates  the  detailed  design  of  the 
CPCs.  The  CDR  (or  series  of  CDRs)  for  a CPCI  is  a technical  review  accomplished 
when  logical  design  is  completed  at  the  level  of  flow  charts  (or  equivalent), 
prior  to  coding  and  testing.  When  a complex  CPCI  is  scheduled  to  reach  any 
given  stage  of  the  design/development/test  process  in  increments  of  CPCs  or 
assemblies  of  CPCs,  the  CDR  is  also  scheduled  in  increments,  and  the  actual 
level  of  design  at  which  reviews  are  scheduled  may  be  adjusted  to  optimize 
the  efficiency  of  the  overall  CDR  for  the  CPCI  as  a whole.  Compared  with  the 
PDR,  the  CDR  provides  a more  detailed  basis  for  verifying  design  integrity 
and  compatibility  with  the  CPCI  Development  Specification.  The  primary 
purpose  of  the  CDR  is  to  establish  the  accomplished  design  and  development  as 
the  basis  for  continuation  of  CPCI  development. 
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Some  coding  may  be  accomplished  prior  to  CDR.  This  is  particularly  true  in 
areas  where  the  requirements  are  straightforward,  where  the  developer  feel^ 
that  coding  is  necessary  to  assure  that  the  program  design  is  feasible,  or 
when  certain  long  lead  time  modules  are  required.  When  the  developer  commits 
design  to  coding,  prior  to  CDR,  he  naturally  accepts  responsibility  for  the 
decision.  When  this  occurs,  and  changes  are  required  as  a result  of  the  CDR, 
the  developer  will  be  required  to  make  the  necessary  adjustments.  The  require- 
ments for  conducting  a CDR  are  specified  in  Appendix  D of  MIL-STD-\.521  A (USAF). 

3.4.1  Materials  to  be  Reviewed 

The  actual  details  of  the  CDR  will  be  initially  defined  in  the  contract  SOW. 

The  data  required  for  CDR  will  be  specified  in  the  CDRL.  The  agenda  for  the 
review  will  consolidate  and  detail  this  information.  The  developer  will 
prepare  the  agenda  and  submit  it  to  the  PO  for  approval.  Basically  the 
technical  information  presented  at  the  CDR  is  the  same  data  contained  in  a 
CPCI  Product  (Part  II)  Specification,  with  the  exception  of  the  program 
listings  and  the  results  of  software  engineering  studies  that  were  conducted 
to  arrive  at  the  CPC-design  decisions  (e.g.,  design  trade  studies  and  analyses) 

Trade  studies  are  accomplished  at  all  levels  from  system  to  detailed  design. 

The  most  significant  trade  studies  are  accomplished  during  the  early  phases 
of  the  system  acquisition  life-cycle.  By  CDR,  the  majority  of  trade  studies 
have  been  completed.  The  trade  studies  accomplished  at  this  time  are  done 
by  the  programmers  as  they  work  out  the  design  of  the  CPCs. 

3.4.2  Requirements 

The  purpose  of  the  CDR  is  to  critically  review  the  detailed  design  of  the  CPCI 
(e.g.,  flow  diagrams,  data,  data  structure,  and  prose  descriptions)  to 
determine  if: 

• The  requirements  of  the  CPCI  Development  Specification 
can  be  implemented. 

• The  detailed  design  is  compatible  with  the  design 
structure  presented  at  the  PDR. 

0 The  detail  is  sufficient  to  initiate  coding. 

The  PO  must  bear  in  mind  that  the  contractor  is  responsible  for  the  design  of 
the  CPCI  until  he  demonstrates  that  the  design  satisfies  the  requirements  of 
his  contract  specification  [CPCI  Development  (Part  I)  Specification].  The  PO 
should  not  insist  on  any  particular  design  for  the  contractor  to  follow.  If 

a specific  design  is  required,  it  should  be  so  stated  in  the  Development 
Specification  as  a design  contraint  to  be  implemented  by  the  contractor.  The 
PO  can  require  corrective  action  if  the  contractor  shows  a lack  of  effective 
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technical  management  capability  or  insufficient  technical  capability.  The 
kinds  of  corrective  action  that  the  PO  can  take  are  (1)  unqualified  approval, 

(2)  approval  with  contingent  action  items,  (3)  approval  with  deviations,  or 
(4)  disapproval  (see  Paragraph  3.3.3). 

Normally  design  reviews  are  conducted  by  having  the  contractor  present  his 
design  and  the  rationale  used  in  arriving  at  his  design  decisions.  The  PO's 
role  is  to  question  the  design  presented  in  an  attempt  to  identify  deficiencies. 
If  the  deficiencies  are  extensive,  the  PO  may  find  the  design  technically 
unacceptable  and  may  reschedule  the  CDR  for  a later  date.  If  the  basic  design 
is  found  to  be  technically  sound,  the  deficiencies  can  be  listed  in  the 
minutes  of  the  CDR  and  follow-up  action  can  be  taken  to  determine  if  the 
deficiencies  were  corrected. 

In  evaluating  the  contractor's  design,  the  PO  should  assure  that: 

• All  outstanding  deficiencies  identified  at  PDR  have 
been  corrected. 

• The  requirements  in  the  Development  Specification  are 
current  and  reflect  approved  ECPs. 

• The  contractor  provides  information  to  demonstrate  that  the 
sizing  and  timing  requirements  can  be  satisfied. 

• The  interfaces  between  the  CPCs  are  identified 
and  compatible. 

• All  requirements  in  the  CPCI  Development  Specification 
can  be  identified  in  specific  portions  of  the  detailed 
design. 

• The  relationship  between  the  CPCs  and  the  data  base 
elements  is  identifiable. 

• The  external  CPCI  interfaces  are  compatible  with  those 
stated  in  the  Development  Specification. 

• The  design  constraints  specified  in  the  Development 
Specification  have  been  implemented. 

• The  design  implements  the  established  programming 
standards  established  in  Section  3.2  of  the  Development 
Specification. 

• The  test  plan  and  procedures  have  been  updated  to 
reflect  changes  due  to  approved  ECPs. 
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• The  test  plan  is  compatible  with  Section  4 of  the  Development 
Specification. 

• The  test  procedures  will  adequately  demonstrate  the 
performance  capabilities  of  the  CPCI. 

3.4.3  Post-Review  Action 

The  contractor  will  publish  the  CDR  minutes,  normally  within  5 working  days 
of  the  completion  of  CDR.  Discrepancies,  identified  during  the  course  of 
the  review,  will  be  reflected  in  the  minutes  along  with  a schedule  for  their 
correction.  The  PO  should  initiate  follow-up  action  to  assure  that  the 
discrepancies  are  corrected. 
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SECTION  4 - CONFIGURATION  MANAGEMENT  AUDITS 


The  purpose  of  configuration  management  audits  is  to  verify  compliance  with 
requirements.  As  stated  in  AFSCP  800-3,  compliance  with  specification  and 
other  contract  requirements  is  verified  by  means  of  configuration  audits.  The 
audits  function  validates  accomplishment  of  development  requirements  and 
achievement  of  a product  configuration  through  the  Cl's  technical  documenta- 
tion. Two  kinds  of  audits  are  performed,  functional  and  physical.  The 
Functional  Configuration  Audit  (FCA)  verifies  that  the  CPCI  performance 
satisfies  the  requirements  of  the  CPCI  Development  Specification.  The 
Physical  Configuration  Audit  (PCA),  on  the  other  hand,  verifies  that  the 
technical  documentation  is  compatible  with  the  CPCI  that  has  successfully 
passed  FCA.  There  is  a shift  in  responsibility  from  conducting  design  reviews 
to  conducting  audits.  In  both  cases,  a joint  PO/developer  team  is  involved. 
For  the  design  reviews  the  co-chairman  with  primary  responsibility  represents 
the  developer.  For  audits  the  co-chairman  with  primary  responsibility 
represents  the  PO.  PO  representation  at  the  reviews  is  normally  from  the 
System  Engineering  Office,  while  PO  representation  at  the  audits  is  normally 
from  the  Configuration  Management  Office.  The  official  documents  stating  the 
requirements  for  conducting  audits  are: 

• Air  Force  In-house  Direction.  Chapter  5 of  AFR  65-3. 

• Contractual  Requj remej^ts . Appendixes  E,  F,  and  G of 
MTL-STD-T5?1A  (USAFT. 

• Additional  Information.  Chapter  5 of  AFSCM/AFLCM  375-7 
and  Chapter  9 of  AFSCP  800-3  (guidance  for  conducting 
FQRs  is  also  addressed  in  this  chapter). 

Both  the  FCA  and  the  PCA  are  discussed  in  the  following  paragraphs  in  terms  of 
(1)  materials  to  be  reviewed,  (2)  requirements,  and  (3)  post-review  activities. 

4.1  FUNCTIONAL  CONFIGURATION  AUDIT 

The  basic  purpose  of  the  FCA  is  to  determine  if  the  CPCI  has  satisfied  the 
requirements  of  the  CPCI  Development  Specification.  This  is  accomplished  by 
auditing  the  results  of  the  CPCI  qualification  tests  to  determine  the 
qualification  status  of  the  CPCI.  When  a CPCI  test  program  has  Preliminary 
Qualification  Tests  (PQTs)  and  a Formal  Qualification  Test  (FQT),  the  FCA  may 
begin  in  increments  with  the  PQTs;  however  the  FCA  cannot  be  considered 
complete  until  FQT  has  been  accomplished.  The  management  and  control  of  the 
test  program  along  with  the  evaluation  of  the  test  results  are  the  responsi- 
bility of  the  Test  Division;  Configuration  Management  is  responsible  for 
auditing  the  results.  Appendix  E of  MIL-STD-1 521 A (USAF)  provides  the 
requirements  for  conducting  FCAs. 


4.1.1  Materials  to  be  Reviewed 


i 
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The  materials  normally  reviewed  at  FCA  include: 

• CPCI  Development  (Part  I)  Specification 

• Final  draft  CPCI  Product  (Part  II)  Specification 

• CPCI  test  plan 

• CPCI  test  procedures 

• CPCI  test  reports  of  all  formal  tests  completed 

• PDR  and  COR  Minutes 

• Configuration  management  status  reports 

• List  of  qualification  tests  yet  to  be  accomplished 

4.1.2  FCA  Requirements 

In  conducting  the  FCA,  the  PO  should: 

• Review  the  configuration  status  records  to  determine 
the  implementation  status  of  all  approved  ECPs. 

• Review  the  Development  Specification  to  determine  if 

it  has  been  updated  to  reflect  approved  ECPs,  including 
both  Sections  3 and  4. 

• Assure  that  there  is  a test  procedure  and  test  report 
for  each  CPCI  qualification  test  identified  in  the  test 
plan.  This  requirement  will  only  exist  if  called  for 
in  the  CDRL.  If  qualification  of  certain  functions  was 
to  be  accomplished  during  Computer  Program  Test  and 
Evaluation  (CPT&E),  determine  if  the  contractor  has 
delivered  the  test  data,  if  it  has  been  evaluated,  and 
if  it  was  sufficient  to  qualify  the  specified  functions. 

• Determine  which  functions,  if  any,  were  to  be  qualified 
during  PQTs  (as  stated  in  Section  4 of  the  Development 
Specification)  and  determine  if  the  requirements  were 
satisfied. 

• Determine  if  the  functions  that  failed  to  qualify  during 
the  PQT  were  satisfied  during  FQT. 

• Assure  that  the  FQT  results  were  documented,  evaluated, 
and  found  to  have  satisfied  the  requirements. 


36 


( - . ■ ■ - 


I 


When  a CPCI  fails  to  meet  requirements  there  are  several  items  that  must  be 
considered  prior  to  deciding  on  a course  of  action. 

• How  critical  are  the  areas  that  have  not  qualified? 

I What  impact  would  the  non-qual ified  areas  have  on  the 
conduct  of  System  DT&E? 

• What  schedule  and  cost  impact,  if  any,  will  delaying  the 
CPCI  have  on  System  DT&E? 

e How  is  the  contractor  held  responsible  for  the 
unqualified  requirements? 

The  courses  of  action  available  to  the  PO,  which  will  be  formally  communicated 
to  the  developer,  are: 

• Unqualified  approval  which  indicates  that  the  CPCI  has 
satisfied  the  performance  requirements  and  that  the  FCA 
has  been  successfully  completed. 

• Approval  with  contingent  action  items  when- the  audit  is 
not  considered  accomplished  until  satisfactory  completion 
of  the  actions.  Normally  these  action  items  must  be 
completed  prior  to  PCA. 

• Approval  with  deviations  when  it  is  in  the  interest  of  the 
program  to  authorize  limited  approval  and  protect  program 
schedules.  In  this  case  the  PO  may  plan  to  proceed  with 
PCA  and  to  have  the  developer  correct  the  deviations  after 
PCA  and  to  audit  their  accomplishment  during  FQR. 

• Disapproval  when  the  audit  is  unsatisfactory  or  when 
contractual  requirements  have  not  been  satisfied. 

Normally  this  action  will  require  modifications  to  the 
design  and  a rescheduling  of  qualification  testing  and 
the  FCA. 

On  occasions,  the  requirements  in  the  Development  Specification  may  be  too 
stringent  and  it  may  be  desirable  to  change  the  specifications  to  agree  with 
CPCI  performance. 


4.1.3  Post-Audit  Action 


Normally,  within  5 working  days,  the  contractor  will  document  the  minutes  of 
the  FCA.  The  PO  Configuration  Manager  will  officially  record  the  results  of 
the  FCA  for  the  purpose  of  communicating  the  results  to  the  Procuring  Contract- 
ing Officer  (PCO).  Requirements  not  scheduled  for  qualification  until  after 
FCA  should  be  listed  on  a DO  Form  250  and  presented  at  PCA.  These  require- 
ments should  be  reviewed  and  audited  at  FQR. 

4.2  PHYSICAL  CONFIGURATION  AUDITS 


The  PCA,  like  the  FCA,  is  a configuration  management  audit.  At  the  FCA,  it 
was  determined  that  the  CPCI  performed  in  accordance  with  the  requirements  in 
the  CPCI  Development  (Part  I)  Specification.  The  objective  of  the  PCA  is 
to  determine  if  the  support  documentation  (i.e..  Product  Specification, 
manuals,  and  handbooks)  accurately  reflects  and  is  compatible  with  the 
qualified  CPCI.  The  PCA  is  normally  held  after  completion  of  FQR  and  FCA. 

As  a result  of  the  PCA,  recommendations  are  made  to  the  responsible  contract 
administration  office  whether  or  not  to  accept  the  CPCI.  When  conducting  a 
PCA,  the  audit  team  members  must  be  aware  of  the  important  role  that  the  CPCI 
Product  (Part  II)  Specification  plays  during  Deployment.  This  specification 
provides  a technical  description  of  the  qualified  CPCI,  i.e.,  it  identifies 
the  exact  configuration  of  the  qualified  CPCI.  It  is  the  prime  instrument 
used  for  making  design  modifications  (software  maintenance)  to  the  CPCI. 

The  Product  Specification  provides  the  technical  description  of  the  qualified 
CPCI.  It  is  therefore  critical  that  the  Product  Specification  accurately 
describes  the  CPCI.  Appendix  F of  MIL-ST.D-l  521A  (USAF)  identifies  the 
requirements  for  conducting  PCAs. 

4.2.1  Materials  to  be  Reviewed 

The  actual  list  of  materials  to  be  reviewed  at  the  PCA  is  identified  when  the 
PCA  agenda  is  approved.  Normally,  these  materials  include: 

0 CPCI  Development  (Part  I)  Specification 

0 Final  draft  of  the  CPCI  Product  (Part  II)  Specification 

0 Version  description  document 

0 Manuscript  copy  of  positional  handbook 

0 Manuscript  of  computer  programming  manual 

0 Manuscript  of  computer  program  user's  manual 

0 FCA  minutes 

0 Configuration  index 

0 Configuration  status  report 
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• CPC  I development  record 

• Proposed  DO  Form  250  (including  shortage  list) 

• List  of  changes  to  the  CPCI  made  during  qualification 
testing 

See  the  Software  Documentation  Requirements  guidebook  for  descriptions  of  the 
documents  listed. 

In  addition  to  the  above  listed  materials  the  following  identification  infor- 
mation, for  the  items  to  be  accepted,  should  be  available. 

• Nomenclature 

• Specification  identification  number 

• CPCI  identifiers 

• Code  identification  number  (H4-1  Number) 

4.2.2  Requirements 

The  PCA  for  a CPCI  is  focused  on  the  Product  Specification  and  the  support 
documentation  package  that  describes  the  specific  version  of  the  qualified 
CPCI.  In  implementing  the  requirements  of  Appendix  F of  MIL-STD-1521A  (USAF), 
the  SD  must  clearly  understand  the  differences  between  hardware  and  software 
PCAs.  These  differences  are  not  always  apparent  in  military  specifications 
and  standards.  The  basic  differences  are  as  follows: 

0 The  Hardware  PCA  is  conducted  on  the  first  production  article 
to  determine  if  the  Production  Specification/drawings  accurately 
describe  the  production  unit.  The  hardware  PCA  supports  a 
major  decision  regarding  production  go-ahead. 

0 The  Software  PCA,  on  the  other  hand,  is  a comparison  of  the 
Product  Specification  and  the  support  documentation  package 
with  the  qualified  CPCI  (a  development  article).  A success- 
ful PCA  for  a CPCI  is  a major  milestone  for  determining 
whether  Full-Scale  Development  contract  products  should  be 
accepted . 

Performing  the  PCA  on  a CPCI  is  a technically  difficult  and  tedious  task  because 
of  the  intricacy  and  volume  of  details.  The  PCA  is  conducted  by  comparing 
the  documentation,  including  the  listings,  with  the  qualified  CPCI.  . Visibility 
(inspection)  is  provided  by  the  listings.  In  theory,  the  steps  are: 

0 Verify  source  code  printouts  used  to  assemble  or 

compile  the  master  tape  against  the  Product  Specification. 

0 Through  detailed  technical  analysis,  verify  the  narrative 
information  and  flowcharts  against  the  listings,  for: 
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- Accuracy 

- Completeness 

- Understandabil ity 

Software  tools  can  be  used  to  aid  the  SD  in  conducting  a PCA.  These  tools  can 
be  used  to  generate  documentation  from  the  qualified  CPCI . Examples  of  such 
tools  are  flow  charts  generated  by  automated  flow-charter  software  packages, 
set-used  matrixes,  and  COMPOOL  outputs  (see  Appendix  A of  the  Verification 
guidebook  for  a description  of  these  tools). 

The  documentation  generated  in  using  these  tools  should  be  compared  to  the 
delivered  documentation  to  determine  its  accuracy.  In  addition  to  making  the 
comparisons,  the  SD  should  assure  that: 

• The  delivered  documents  satisfy  the  format  and  contents 
specified  in  the  Data  Item  Description  (DID)  and  are 
responsive  to  the  CDRL. 

• The  CPCI  Product  (Part  II)  Specification  is  accurate, 
complete,  and  compatible  with  the  CPCI. 

• The  CPCI  Development  (Part  1)  Specification  has  been 
maintained  to  reflect  all  approved  ECPs. 

• The  configuration  management  status  accounting  records 
provide  complete  traceability  and  document  the  status 
of  ECPs  and  the  CPCI  configuration. 

• The  version  description  document  accurately  describes 
the  delivered  version  of  the  CPCI. 

• The  CPCI  Product  Specification  reflects  the  currently 
approved  configuration  of  the  CPCI. 

• The  user/operator  manuals  have  been  validated.* 


*The  positional  handbooks,  user  manuals,  and  operator  manuals  should  be 
validated  prior  to  System  DTiiE.  The  verification  of  these  manuals  will 
normally  be  accomplished  during  System  DT&E.  The  definitions  for  valid- 
ation and  verification  used  herein  are  in  accordance  with  APR  8-2,  i.e.: 

• Validation  is  the  process  by  which  the  contractor  tests 
operating  and  maintenance  procedures  for  technical  accuracy 
and  adequacy. 

• Verification  is  the  process  by  which  operating  and 
maintenance  procedures  are  tested  and  proven  under 
Air  Force  jurisdiction. 
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4.2.3  Post-Audit  Actions 


The  contractor  will  publish  the  PCA  minutes,  normally  within  5 working  days 
after  completion  of  the  PCA. 

The  PO  notifies  both  the  contractor  and  the  PCO  of  any  post-audit  action, 
normally  within  10  working  days  of  the  receipt  of  the  PCA  minutes. 

The  PCO  will  sign  the  DO  Form  250  indicating  contractual  acceptance  of  the 
CPCI  and  related  products.  If  the  DO  Form  250  has  shortages  listed,  then 
the  signed  DO  250  represents  a conditional  acceptance  until  the  shortages 
have  been  satisfied.  When  test  and  evaluation  results  are  a condition  of 
acceptance  and  are  not  available  prior  to  PCA,  the  following  note  is 
normally  extended  if  conditional  acceptance  is  made  by  the  PO: 

"Aaaeptanae  and  payments  are  contingent  upon  receipt  of  approved 
test  and  evaluation  results."  A conditional  acceptance,  with 
outstanding  qualification  requirements,  normally  requires  an  FQR. 

4.3  FORMAL  QUALIFICATION  REVIEW 

The  FQR  is  that  milestone  within  the  system  acquisition  life  cycle  when  the 
Government  officially  certifies  that  the  CPCI  is  qualified.  The  actual 
scheduling  of  the  FQR  will  depend  on  the  System/CPCI.  If  a CPCI  can  be 
qualified  at  FCA  then  the  FQR  will  be  scheduled  as  part  of  FCA.  If  a CPCI 
has  requirements  yet  to  be  satisfied  at  the  time  the  PCA  is  conducted,  then 
an  FQR  will  be  scheduled  and  conducted  after  PCA,  normally  during  System  DT&E. 

The  requirements  for  the  conduct  of  the  FQR,  the  materials  to  be  reviewed,  and 
the  post-review  actions  are  essentially  the  same  as  those  for  FCA.  Appendix  G 
of  MIL-STD-1 521A  (USAF)  specifies  the  requirements  for  conducting  an  FQR. 
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SECTION  5 - FCA/FQR  AND  PCA  SAMPLE  FORMS 


This  section  contains  modified  sample  forms  from  MIL-STD-1 521A  (USAF)  that  may 
be  used  in  conjunction  with  design  reviews  and  configuration  management  audits 
These  are  not  official  forms  jut  are  presented  as  a recommended  way  of  identi- 
fying and  recording  critical  data. 

5.1  ACTION  ITEM  FORM 


The  Action  Item  Form  is  used  by  the  PO  to  further  define  the  action  items 
identified  in  the  minutes  of  the  reviews/audits  and  to  assign  a control 
number  and  responsible  individual  to  track  and  monitor  the  action.  The 
responsible  individual  will  report  the  status  of  the  action  item  and  will 
coordinate  the  approval  of  the  end  result.  For  each  action  item  identified 
in  the  minutes  an  action  item  form  will  be  initiated.  The  Action  Item  Form 
is  illustrated  in  Figure  2. 


5.2  CERTIFICATION  ATTACHMENT  FOR  FCA/FQR 


The  Certification  attachment  for  FCA/FQR  is  used  to  record  the  necessary  data 
leading  up  to  the  qualification  and  certification  of  a Cl.  This  record  is 
developed  during  FCA  and  is  finalized  when  the  FQR  is  conducted.  The  forms  are 
completed  by  the  FCA  and  FQR  team  members  and  are  illustrated  in  Figure  3. 

5.3  CERTIFICATION  ATTACHMENT  FOR  PCA 


The  certification  attachment  for  the  PCA  is  used  by  the  PCA  team  to 
records  during  PCA.  The  sample  format  for  the  PCA  certification  is 
in  Figure  4. 


establ ish 
illustrated 
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ACTION  item 


ACTION  required/compliance 


FOLLOW  UP 
STATUS: 


ASSIGNEE 


PHONE  AGENCY 


DUE  DATE 


DATE 

COMPLETED: 

DOCUMENT 

NO. 

COORDINATORS 

TECHNICAL  APPROVAL 

PHONE 

AGENCY 

SIGNATURE 

DATE 

HIONE 

AGENCY 

SIGNATURE 

EATE 

SIGNATURE 


PROCURINO 

ACTIVITY 


QftTE  CONTRACTOR 


mTE  CONTRACTS 


Figure  2.  Sample  Action  Item  Form 
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FCA  CHECK  SHEET  FOR  CPCIs 


Nomenclature  

CPCI  No. Date 

Contractor  Requirements  Yes  No 

1.  CPCI  Development  Specification  Available  

2.  Final  draft  CPCI  Product  Specification  Available  

3.  CPCI  Qualification  (Category  1)  Test  Plan  Approved  

4.  Qualification  (Category  1)  Test  Procedures 

Submitted  and  Approved  

5.  Qualification  Test  Complete  

6.  Qualification  Test  Results  Compiled  and  Available  

7.  Qualification  Test  Results  Reviewed  

8.  Qualification  Testing  Witnessed  

9.  Qualification  Test  Report(s)  Available 


Comments 


Figure  3.  Sample  Certification  Attachment  (1  of  7) 


FUNCTIONAL  CONFIGURATION  AUDIT  (FCA) 


FOR 


Cl  NO.(s) 

CONTRACT  NO. 


PRIME  CONTRACTOR: 


EQUIPMENT  MANUFACTURERS: 


APPROVED  BY 

(CONTRACTOR) 


APPROVED  BY 

(PROCURING  ACTIVITY) 


DATE  DATE 


DEFINITIONS: 

COMMENT:  A note  explaining,  Illustrating,  or  crltlzing  the  meaning  of  a 
writing.  Items  of  this  nature  should  be  explored  by  the  contractor  and/ 
or  the  Procuring  Activity,  but  corrective  action  is  NOT  necessary  to 
successfully  accomplish  a FCA. 

DEFICIENCY:  Deficiencies  consist  of  two  types:  (1)  conditions  of  charac- 
teristics in  any  hardware /software  which  are  not  in  compliance  with  speci- 
fied configuration,  or  (2)  Inadequate  (or  erroneous)  configuration,  identi- 
fication which  has  resulted,  or  may  result  in  configuration  items  that  do 
not  fulfill  approved  operational  requirements. 


Figure  3.  Sample  Certification  Attachment  (2  of  7) 
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A 


SCOPE/PURPOSE 


Scope; 

Functional  Configuration  Audit  (FCA)  was  conducted  on  the  following 
end  item: 

CPCI  No.  Nomenclature  Version  No. 


1 


PURPOSE:  The  purpose  of  this  FCA  was  to  verify  that  the  CPCI(s)  perform- 
ance complies  with  the  Part  I Development  Specification. 


Figure  3.  Sample  Certification  Attachment  (3  of  7) 


1.  V 


i _ 
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j 


Contract: 


FUNCTIONAL  CONFIGURATION  AUDIT 


(;:ertification  sheet  no.  i 

(equipment/computer  programs) 


Contractor: 


Cl  No.: 


Qualification  Test  Procedvu~es  and  Results.  The  qualification  test/ 
analysis  results  have  been  reviewed  to  insure  that  testing  is  adequate, 
properly  done,  and  certified.  (All  test  procedures  and  Interface  docu- 
ments shall  be  reviewed  to  assure  that  the  documents  have  been  approved 
by  the  Procuring  Activity.  All  test  data  sheets  shall  be  reviewed  to 
assure  that  the  test  was  witnessed  by  a representative  of  DCAS/AFPFO  or 
the  Procuring  Activity. ) 

Attached  is  a list  of  the  documents  reviewed. 

Check  One 

Procedures  and  results  reviewed  satisfy  the  requirements  and  are 
accepted.  See  Attachment  for  comments. 

Attached  is  a list  of  deficiencies. 


Signature(s)  of  FCA  Team  Member (s) 


•Sub-Team  Chairperson 


Figure  3.  Sample  Certification  Attachment  (4  of  7) 
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SPECIFICATIOW/TESTIIIG  REVIEW 


Nomenclature 


Specification  No. 


Test  Procedures 


Test  Result 


Figure  3.  Sample  Certification  Attachment  (5  of  7) 
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Figure  3.  Sample  Certification  Attachment  (6  of  7) 


FORMAL  QUftLIFICATICW  REVIEW 
(For  Equipment/Computer  Programs) 


Contract: Date 


Contractor: 


Cl  No.: 


Formal  Qualification  Review.  Qualification  Test/Analysis  results  have 
been  reviewed  to  verify  that  the  actual  performance  of  the  Cl/CPCI  com- 
plies with  its  development  specification  and  that  sufficient  test  results 
are  available  to  insure  the  Cl/CPCI  will  perform  in  its  system  environ- 
ment . 

Attached  is  a list  of  the  documents  reviewed. 

Check  One 

Results  reviewed  satisfy  PQR  requirements  and  the  Cl/CPCI  is  quali- 
for  entry  into  the  Government  Inventory. 

Results  reviewed  are  unsatisfactory/insufficient  for  PQR.  PQR  will 
be  delayed  until  it  is  determined  that  siifficient  information  on  the 
Cl/CPCI's  Qualification  is  available. 


Signature(s)  of  PQR  Team  Member(s) 


V 


' V c 

SAMPI£  CERTIFICATION  ATTACHMENT 
SAMPLE  PCA  CHECKLIST 

The  foilowin^j  hardware,  computer  program(s),  documentation  shell  be 
available,  and  the  following  tasks  shall  be  accomplished  at  the  PCA. 

Xj 

Hardware : 


Doc  umen  ta  t i on : Yes  No 

(1)  Approved  final  draft  of  the  Cl  product  specification.  

(2)  A list  delineating  both  approved  and  outstanding  

changes  against  the  Cl. 

(j)  Complete  shortage  list.  

*(i*)  Acceptance  test  procedures  and  associated  test  data  

Engineering  Drawing  Index.  

*(6)  Operating,  maintenance,  and  illustrated  parts 

breakdown  manuals  

*(7)  List  of  approved  material  review  board  actions  on 

waivers.  

^8)  Proposed  DD  Form  250,  "Material  Inspection  and 

Receiving  Report."  

*(9)  Approved  nomenclature  and  nameplates.  

(10)  Manuscript  copy  of  all  CPCI  handbooks/manuals.  

Ql)  Computer  program  version  description  document.  

(12)  Current  set  of  listings  and  updated  flow  charts 

for  each  CPCI.  

(13)  FCA  minutes  for  each  Cl.  


*Does  not  apply  to  CPCIs. 

Figure  4.  Certification  Attachment  for  PCA  (1  of  20) 
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SAMPI£  PCA  CHECKLIST (CONTimjED) 


Tasks : Yes 

(1)  Define  Product  Baseline.  

(2)  Specification  Review  and  Validation.  

*(3)  Drawing  review.  

*(4)  Review  acceptance  test  procedures  end  results.  

(5)  Review  shortages  and  unincorporated  design  changes.  

*(6)  Review  deviations/waivers.  

(7)  Examine  proposed  DD  250.  

*(8)  Review  contractors  Engineering  Release  and  Change 

Control  System.  

*(9)  Review  system  allocation  document.  

(10)  Review  Positional  Handbooks,  Computer  Program 

User's  Manuals,  and  Computer  Programming  Manuals.  

(11)  Review  CPCI's  for  the  following: 

(a)  Computer  Program  Component  (CPC)  descriptions  

and  flow  charts. 

(b)  CPC  interface  requirements.  

(c)  Data  base  characteristics,  storage  allocation 

charts  and  timing  and  sequencing  characteris- 
tics.   

*(12)  Review  packaging  plan  and  requirements.  

(13)  Review  status  of  Rights  in  Data.  

(14)  Review  CM  Records  and  Status. 

(15)  Review  FCA  Minutes  Action  Item. 

(16)  Check  compatibility  of  technical 

documentation  with  qualified  CPCI.  ~ 

*Does  not  apply  to  CPCIs. 


Figure  4.  Certification  Attachment  for  PCA  (2  of  20) 
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i.XWlVN'r  - A note  t'xpliiinitv,,  i liustratin*?,  or  criticizing  the  meaning 
ot'  H wrltitu-;.  Item;  of  thin  nature  should  he  explored  by  the  con- 
tractor and/or  the  i'rocuring  Activity,  but  corrective  action  is  NOT  neces- 
sary to  succenafully  aocomplinh  a IXIA . -J 


Dli'CIvi  I'ANOT  - A ivote  explaining,  llluatrating,  or  criticizing  the  differ- 
ence between  writing;..  A note  allowing  the  variance  between  what  exists 
and  what  is  acceptable.  Items  of  this  nature  shall  be  rectified  by  the 
oontract.oi-  prior  to  successful  accomiilisliment  of  a WA. 


SCOPE/PURPOSE 

A Physical  Configuration  Audit  (PCA)  was  conducted  on  the  following  CPCIs; 
CPC I NO.  NOMENCLATURE  VERSION  NO. 


The  purpose  of  the  PCA  was  to  insure  accuracy  of  the  identifying  documen- 
tation and  to  establish  a product  baseline. 


Figure  4.  Certification  Attachment  of  PCA  (5  of  20) 
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I 


-ij 


J 


r 


PHYSICAL  gON?'IGlJRATION  AUDIT 

CERTIFICATION  SHEET  NO.  1 
(tor  equipment/computer  programs) 


Cor.-ract:  AF Date 

Cor.':!  actor;  


Product  Baseline.  The  following  documents  of  the  issue  and  date  shown, 
comprise  the  Product  Baseline  for  the  listed  equipments/computer  pro- 
grams ; 


***ASSE;.!BLy 

TOP 

COMP  PRGM 

SPr'C  NO.  DRAWING 

NO. 

ISSUE 

NOMENCIATUPE 

rignature(s)  of  iYA  Team  Member(s) 


♦ 


***  Not  Applicable  to  CPCIs 
**  Team  Chairperson 
*■  Si'i  -Team  Chairperson 


Figure  4.  Certification  Attachment  of  PCA  (6  of  20) 
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PHYSICAL  CONFIGURATION  AUDIT 


CERTIFICATION  SHEET  NO.  2 
(for  computer  programs! 

Contract;  AF Date 

Contractor;  


t 

f 

i 

[ 

r 


Specification  Review  and  Validation.  Specifications  have  been  reviewed 
and  validated  to  assure  that  they  adequately  define  the  CPCI. 

( 


Check  One 


The  Type  C/Phrt  II  Specifications  are  complete  and  ade- 
quately define  the  Cl.  They  shall,  therefore,  constitute 
the  Product  Baseline.  See  attachment  for  conmentc. 

The  Type  C/Part  II  SjJeclflcatlons  are  unacceptable. 
Attached  is  a list  of  discrepancies. 


Signatures  of  PCA  Team  Members 

* 


*Sub-Team  Chairperson 


Figure  4.  Certification  Attachment  of  PCA  (7  of  20) 
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I 


i. 

I 

i 


1 


A . Specification  Review  and  Validation  Instructions.  The  detailed 
specifications  listed  In  paragraph  B.  below  shall  be  reviewed  for  com- 
pl  .ance  with  the  s licable  requirements.  FJach  specification  shall 
serve  as  the  basic  document  for  configuration  control  of  the  subject 
items.  The  information  contained  within  the  specifications  shall  be 
audited  at  the  PCA. 


B.  Review  and  Validation  Results; 

1.  Specifications  Reviewed  and  Validated 

COMP  P(a4 

SPEC.  NO.  PARAGRAPH  NO.  DATE  NOMENCLATURE 


Cl  NO. 


2.  Specifications  Reviewed  and  Disapproved: 
(Provide  attachment  for  causes.) 


Figure  4.  Certification  Attachment  of  PCA  (8  of  20) 
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Inspection  and  Receiving  Report,"  and  other  records  have  been  reviewed. 


Review  of  Shortages  and  unincorporated  Design  Changes.  All 
shortages  and  unincorporated  design  changes  listed  on  the  pro- 
posed DD  Form  250,  "Material  Inspection  and  Receiving  Report", 
shall  be  reviewed  by  the  Procuring  Activity  or  their  designated 
representatives  for  a determination  of  what  changes  should  be 
accomplished  in  the  field  and  what  changes  should  be  accomplished 
at  the  contractor's  facility.  The  Procuring  Activity  shall  also 
determine  if  the  reported  shortages  and  ml-ncorporated  changes 
are  complete. 


Results . List  the 
that  were  reviewed 


shortages  and  unincorporated  design  changes 
in  compliance  with  requirements. 


Figure  4.  Certification  Attachment  for  PCA  (10  of  20) 


PHYSICAL  CCTIFIGURATION  AUDIT 

CERTIFICATION  SHEET  NO.  6 
(for  equipment /computer  programs) 

AF  Date 


;! 


Review  Devlatlons/Walvers.  A review  of  all  devlatlons/walvers  to  mili- 
tary specifications  and  standards  that  have  been  approved.  The  purpose 
is  to  determine  the  extent  to  which  the  equipment(s)  undergoing  PCA  vary 
from  applicable  specifications  and  standards  and  to  form  a basis  for 
satisfactoiry  compliance  with  these  specifications  and  standards. 

In  accordance  with  this  paragraph,  all  applicable  deviations/waivers 
have  been  reviewed  with  the  following  results: 

Check  One 

The  equipment(s)/computer  program(6)  listed  on  Certification 
Sheet  No.  1 of  this  report  complies  with  all  applicable  sped 
fications  and  standards.  (See  Attachment for  comments). 

Attachment is  a list  of  discrepancies  and/or  comnents. 

Signature(s)  of  PCA  Team  Member(s). 

» 


* Sub-Team  Chairperson 

Figure  4.  Certification  Attachment  of  PCA  (11  of  20) 


Contract: 

Contractor: 
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Deviation/Waiver  Review  Team  Instruction.  All  approved  waivers 
and  deviations  to  military  specifications  and  standards  shall 
be  reviewed  and  recorded.  Also,  record  any  pert  of  the  PGA 
which  fails  to  meet  specifications  or  standards  but  is  not  an 
approved  waiver/deviation. 


Results  of  Team  Review.  List  the  deviations/waivers  against 
the  equipment/computer  programs  being  PCA's  that  were  reviewed. 


Figure  4.  Certification  Attachment  of  PCA  (12  of  20) 


PHYSICAL  CONFIGURATION  AUDIT 


CERTIFICATION  SHEET  NO.  7 
(for  equipment/ccMiputer  programs) 


Contract:  AF Date 

Contractor: 


Examination  of  the  Proposed  DP  230 « The  DD  Form  250  has  been  examined 
to  insure  that  it  adequately  defines  the  equipment/computer  programs 
and  that  unaccomplished  tasks  are  Included  as  deficiencies. 

Check  One 


The  DD  Form  250  adequately  defines  the  equipment/computer 
program  and  all  unaccomplished  tasks  are  Included  as  de- 
ficiencies. 

Attachment is  a list  of  discrepancies  and/or  comments. 


Slgnature(s)  of  PCA  Team  Member(s) 

* 


^•Sub-Team  Chairperson 


Figure  4.  Certification  Attachment  of  PCA  (13  of  20) 


A . Examination  of  the  Proposed  DP  Form  250«  The  proposed  DD  Form 
250  shall  be  examined  for  completeness  and  an  accurate  defini- 
tion of  the  equipment/computer  programs.  Unaccomplished  tasks, 
shortages,  and  certain  specified  discrepancies  uncovered  at  the 
PCA  shall  be  included  in  the  DD  Form  250.  If  the  equipment/ 
computer  programs  is  to  be  shipped  from  the  plant,  the  Program 
Office  representative  will  recommend  to  the  CAO  that  the 
DD  Form  250  be  executed  in  accordance  with  the  terms  of  the 
contract. 
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PHYSICAL  CONFIGURATION  AUDIT 

CERTIFICATION  SHEET  Na  IQ 
(FOR  COMPUTER  PROGRAMS) 


CONTRACT:  AF  DATE 

CONTRACTOR:  


REVIEW  POSITIONAL  HANDBOOKS,  COMPUTER  PROGRAM  USER'S  MANUALS, 
AND  COMPUTER  PROGRAMMING  MANUALS. 


(A)  POSITIONAL  HANDBOOKS 

(B)  COMPUTER  PROGRAMMER  USER'S  MANUAL 

(C)  COMPUTER  PROGRAMMING  MANUALS 


CHECK  ONE 

COMPLETE 

& 

ADEQUATE  , UNACCEPTABLE 


FOR  UNACCEPTABLE  PRODUCTS  SEE  ATTACHED  LIST  OF  DISCREPANCIES 


SIGNATURES  OF  PCA  TEAM  MEMBERS 


*SUB-TEAM  CHAIRPERSON 


Figure  4.  Certification  Attachment  of  PCA  (15  of  20) 


PHYSICAL  CONFIGURATION  AUDI! 

CERTIFICATION  SHEET  NO.  11 
(FOR  COMPUTER  PROGRAMS) 


’I 


CONTRACT:  AF  DATE 

CONTRACTOR:  


REVIEW  CPCI  PRODUCT  SPECIFICATION  FOR  THE  FOLLOWING: 

(A)  COMPUTER  PROGRAM  COMPONENTS  (CPC) 

DESCRIPTIONS  AND  FLOW  CHARTS 

(B)  CPC  INTERFACE  REQUIREMENTS 

(C)  DATA  BASE  CHARACTERISTICS,  STORAGE  ALLOCATION 
CHARTS  AND  TIMING  AND  SEQUENCING  CHARACTERISTICS 


(A) 

(B) 

(C) 


CPC  DESCRIPTIONS  AND  FLOW  CHARTS 
CPC  INTERFACES 

DATA  BASE,  STORAGE  AND  TIMING  INFORMATION 


CHECK  ONE 

COMPLETE 

& 

ADEQUATE  , UNACCEPTABLE 


FOR  UNACCEPTABLE  PRODUCTS  SEE  ATTACHED  LIST  OF  DISCREPANCIES 
SIGNATURES  OF  PCA  TEAM  MEMBERS, 


*SUB-TEAM  CHAIRPERSON 

Figure  4.  Certification  Attachment  of  PCA  (16  of  20) 
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PHYSICAL  CONFIGUR'VTION  AUDIT 

CERTIFICATION  SHEET  NO.  1 3 
(FOR  COMPUTER  PROGRAMS) 


CONTRACT:  AF 

CONTRACTOR:  

t 

I 

REVIEW  STATUS  OF  RIGHTS  IN  DATA 
^HE£K_ONE 

NO  RIGHTS  IN  DATA  PROBLEM  EXIST  WITH  THIS  CPCI 
ATTACHMENT  SPECIFIES  RIGHT  ON  DATA  RESTRICTIONS 

SIGNATURES  OF  PCA  TEAM  MEMBERS 


★ 


*SUB-TEAM  CHAIRPERSON 


V. 

Figure  4.  Certification  Attachment  of  PCA  (17  of  20) 
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CERTIFICATION  SHEET  NO. 14 


r 

I 


mi 

TrOR' COMPUTET?  PROG'RAHS) 


CONTRACT:  AF  DATE 

CONTRACTOR:  


REVIEW  CM  RECORDS  AND  STATUS 

THE  CM  STATUS  ACCOUNTING  RECORDS  HAVE  BEEN 
REVIEWED  AND  FOUND  TO  BE  UP-TO-DATE  AND  ACCURATE. 


(A)  CONFIGURATION  INDEX  (COMPUTER  PROGRAM) 

(B)  CHANGE  STATUS  REPORT 

(C)  VERSION  DESCRIPTION  DOCUMENT 


COMPLETE 

& 

ADEQUATE  , UNACCEPTABLE 


FOR  UNACCEPTABLE  PRODUCTS  SEE  ATTACHED  LIST  OF  DISCREPANCIES 

SIGNATURES  OF  PCA  TEAM  MEMBERS 


♦SUB-TEAM  CHAIRPERSON 


Figure  4.  Certification  Attachment  of  PCA  (18  of  20) 
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PHYSICAL  CONFIGURATION  AUDIT 

CERTIFICATION  SHEET  NO. 15 
(FOR  COMPUTER  PROGRAMS) 


CONTRACT;  AF  DATE 

CONTRACTOR;  


REVIEW  FCA  MINUTES  ACTION  ITEMS.  DETERMINE  THE  DISPOSITION 
OF  ALL  ACTION  ITEMS  LISTED  ON  THE  FCA  MINUTES. 


CHECK  ONE 


ALL  ACTION  ITEMS  HAVE  BEEN  CLOSED  SATISFACTORILY.  OUTSTANDING 
ACTION  ITEMS  ARE  SCHEDULED  FOR  QUALIFICATION  POST  PCA  AND  AN 
FQR  IS  SCHEDULED  FOR  AUDITING  THE  RESULTS. 

ATTACHMENT IS  A LIST  OF  OUTSTANDING  ACTION  ITEMS  ALONG 

WITH  CURRENT  STATUS 


SIGNATURES  OF  PCA  TEAM  MEMBERS 


*SUB-TEAM  CHAIRPERSON 


Figure  4.  Certification  Attachment  of  PCA  (19  of  20) 
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PHYSICAL  CONFIGURATION  AUDIT 

CERTIFICATION  SHEET  NO. 16 
(FOR  computer  PftOOftAMs)  ' 


CONTRACT;  AF  DATE 

CONTRACTOR:  


CHECK  COMPATIBILITY  OF  TECHNICAL  DOCUMENTATION  WITH  QUALIFIED  CPCI 


CHECK  ONE 

ALL  TECHNICAL  DOCUMENTS  ARE  COMPATIBLE  WITH  THE  CPCI  AND 
ARE  FOUND  TO  BE  ACCEPTABLE. 

ATTACHMENT  LISTS  THE  TECHNICAL  DOCUMENTS  FOUND  TO  BE 

ACCEPTABLE  ALONG  WITH  DESCRIPTION  OF  DISCREPANCIES 


SIGNATURES  OF  PCA  TEAM  MEMBERS 


*SUB-TEAM  CHAIRPERSON 


Figure  4.  Certification  Attachment  of  PCA  (20  of  20) 
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SECTION  6 - REVIEWS  AND  AUDITS  PROBLEM  AREAS 


This  section  identifies  some  of  the  more  common  problems  created  by  improper 
implementation  of  reviews  and  audits  requirements.  In  addition,  it  discusses 
how  engineering  design  reviews  can  be  used  to  prevent  other  common  and 
serious  problems. 

6.1  RESPONSIBILITY  AND  AUTHORITY 

PO  personnel  must  have  a clear  understanding  of  their  responsibility  and 
authority  during  the  conduct  of  an  engineering  design  review.  This  is  parti- 
cularly true  during  PDR  and  CDR.  Because  many  design  review  participants 
are  technically  qualified  to  design  the  CPCI,  there  is  a tendency  during 
the  review  to  suggest  design  improvements.  However,  the  Air  Force  is  buying 
a CPCI  that  performs  in  accordance  with  a CPCI  Development  (Part  I)  Specifi- 
cation. The  contractor  is  responsible  for  the  CPCI  design.  If  there  are 
design  defects,  PO  personnel  are  obligated  to  identify  these  defects  to  the 
contractor,  but  the  design  solution  is  the  responsibility  of  the  contractor. 

The  concern  here  is  that  the  PO  should  not  "constructively  change  the  contract" 
by  identifying  a new  design  that  the  contractor  should  implement.  When  a 
contractor  is  directed  to  implement  a specific  design,  the  design  may  be  in 
conflict  with  the  performance  requirements  specified  in  the  CPCI  Development 
Specification.  When  this  occurs  the  contractor  may  be  legally  relieved  of 
his  responsibility  to  satisfy  the  performance  requirements. 

6.1.1  Requirements 

Prior  to  conducting  engineering  design  reviews,  the  review  team  should  be 
briefed  on  its  responsibilities.  The  PO  should  identify  defects  and  allow 
the  contractor  to  provide  the  solutions.  Suggestions  may  be  offered,  but 
they  should  be  clearly  identified  as  such.  (See  "ESD  Program  Manager's 
Handbook,"  Chapter  19,  for  a discussion  of  the  theory  of  constructive  change.) 

6.2  CPCI  DEVELOPMENT  (PART  I)  SPECIFICATION 

One  of  the  most  serious  problems  encountered  in  the  acquisition  of  CPCIs,  is 
the  inadequacy  of  the  CPCI  Development  Specification.  Major  causes  of  this 
problem  include  the  contractor's  lack  of  expertise  in  Development  Specifica- 
tion preparation  and  the  PO's  lack  of  technically  qualified  review  personnel. 
Preparation  of  a Development  Specification  requires  system  engineering  expertise 
as  well  as  knowledge  of  the  operational  aspects  of  the  system.  The  software 
contractor  often  assigns  software  experts  to  perform  this  task  and  the  resul- 
tant requirements  are  usually  stated  in  data  processing  terms  rather  than 
system  performance  (operational)  terms.  This  results  in  an  incomplete 
specification  written  at  the  wrong  level.  This  problem  is  further  compounded 
when  the  PO  does  not  allocate  sufficient  time  for  the  contractor  to  generate 
the  specification. 


6.2.1  Recommendations 


When  conducting  the  SRR  and  the  SDR,  the  PO  should  determine  the  capability 
and  experience  of  the  contractor's  technical  team.  During  the  generation  of 
a CPCI  Development  Specification  the  contractor  should  be  using  his  system 
engineering  organization  to  define  requirements.  This  organization  should  be 
supported  by  data  processing  experts  whose  job  is  to  determine  the  feasibility 
of  the  stated  requirements.  In  reviewing  the  performance  requirements  at  SRR 
and  SDR  the  PO  should  assure  that: 

• Requirements  are  stated  in  performance  terms. 

t CPCI  interfaces  are  identified  and  specified. 

• The  functional  relationships  have  been  identified. 

• Only  the  necessary  design  constraints  are  identified. 

See  the  Computer  Program  Development  Specification  guidebook  for  further 
discussion  of  contents. 

Any  time  there  is  a question  regarding  the  adequacy  of  the  Development  Speci- 
fication it  is  better  t delay  authentication  until  the  inadequacies  have 
been  corrected.  This  is  especially  true  for  a contract  for  Full-Scale 
Development  which  absorbs  the  Validation  Phase  effort  at  the  front  end. 

In  this  case,  there  may  be  a change  of  the  contract  specification  from  the 
System/System  Segment  Specification  to  the  individual  CI/CPCI  Development 
(Part  I)  Specification.  It  is  essential  that  these  be  sound  documents. 

Many  ESD  contracts  attempt  to  dispose  of  this  problem  of  changing  baselines 
by  inserting  a statement  in  the  contract  such  as  "notwithstanding  any  other 
provisions  of  this  contract  the  contractor  is  not  relieved  from  meeting  the 
performance  requirements  in  the  System  Specification."  The  fact  that  this 
statement  is  embedded  in  the  contract  does  not  necessarily  hold  the  contractor 
to  the  System  Specification.  If  during  the  execution  of  the  contract  the 
Government  takes  steps  to  authenticate  (approve)  the  Cl  Development  Specifi- 
cation, this  action  may  make  the  statement  null  and  void.  When  the  Govern- 
ment approves  a specification,  that  specification  becomes  a Government  specifi- 
cation and  the  contractor's  responsibility  is  to  implement  the  requirements 
of  the  approved  specification.  Faced  with  this  situation,  PO  personnel 
should  seek  guidance  from  the  Procurement  Office. 

6.3  SCHEDULING  FOR  PDR 

The  PDR  should  be  conducted  when  the  contractor  is  prepared  to  present  his 
overall  CPCI  design.  Often  the  schedule  and  agenda  for  PDR  are  not  appropri- 
ate for  this  objective.  In  some  cases  the  PDR  is  used  as  a review  session 
for  the  Development  (Part  I)  Specification.  If  the  Development  Specification 
was  not  authenticated  prior  to  PDR,  then  the  objective  of  reviewing  the 
CPCI  cannot  be  met  because  the  design  must  be  based  upon  the  Development 
Specification.  Also,  PDR  is  customarily  held  60-90  days  after  start  of  the 
Full-Scale  Development  Phase.  This  may  not  be  sufficient  time  to  prepare  the 
design  (and  required  trade  studies)  for  a complex  CPCI. 


6.3.1  Recommendations 


The  contractor  should  be  allowed  to  propose  a PDR  schedule  consistent  with  the 
overall  design  requirements  of  the  CPCI.  The  Development  Specification  should 
be  authenticated  at  the  beginning  of  the  Full-Scale  Development  Phase.  At 
most,  the  PDR  should  be  used  to  review  information  that  will  replace  TBDs  in 
the  Development  Specification.  The  use  of  TBDs  at  the  time  of  authentication 
should  be  minimized  and  localized  to  specific  interface  contents  and  formats 
that  are  not  known  at  the  time  of  authentication.  The  specification  should 
not  have  whole  sections  marked  TBD.  Furthermore,  the  TBDs  should  be  limited 
so  that  when  the  information  is  supplied,  it  will  not  impact  contract  costs 
or  schedules.  Whenever  TBDs  are  used,  specific  responsibility  and  schedules 
should  be  assigned  for  supplying  the  required  information. 


6.4  SCHEDULING  OF  CDR 

CDR  schedules  are  often  rushed.  When  this  happens  insufficient  design  is 
performed  prior  to  CDR.  Although  the  insufficient  design  is  not  always 
apparent  during  CDR,  it  becomes  obvious  sometime  after  coding  begins  and 
rework  is  required. 

6.4.1  Recommendations 

Assure  that  the  contractor  has  all  required  data  available  for  the  design 
reviews.  If  it  is  not  available,  reschedule  the  design  review  meeting  or 
review  the  available  data  and  identify  the  missing  information  in  the  minutes. 
The  contractor  should  be  required  to  schedule  the  presentation  of  the  missing 
information  and  the  PO  should  then  plan  a follow-up  design  review  meeting. 
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APPENDIX  A - GLOSSARY 
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This  appendix  consists  of  (1)  definitions  of  major  terms  used  throughout  this 
guidebook  and  (2)  abbeviations  used  herein. 


DEFINITIONS 

Authenticate.  Approval  signature  by  a responsible  person  of  the  procuring 
activity.  Authentication  by  the  procuring  activity  normally  will  be 
accomplished  on  that  issue  of  the  specification  which  is  to  be  the  con- 
tractual requirement  for  the  baseline  which  that  particular  specifica- 
tion defines  [MIL-STD-483  (USAF)]. 

Certification.  To  verify  that  the  requirements  have  been  satisfied.  Appendix  G 
of  MIL-STD-1 521A  (USAF)  refers  to  the  Government  officially  recognizing 
that  the  Cl  has  been  qualified  and  certifying  to  that  effect.  The  use  of 
the  word  certification  herein  is  in  accordance  with  the  Webster  definition. 
The  use  of  the  term  certification  in  this  guidebook  differs  from  the  use 
within  the  "Validation  and  Certification  Guidebook." 

Computer  Program  Configuration  Item  (CPCI).  A computer  programming  end  product 
whose  development  and  subsequent  modification  is  subject  to  configuration 
management. 

Computer  Programming  Test  and  Evaluation.  Tests  conducted  prior  to  and  in 
parallel  with  preliminary  or  Formal  Qualification  Tests.  These  tests 
are  oriented  primarily  toward  the  support  of  the  contractor's  design  and 
development  process. 

Configuration  Item  (Cl).  An  aggregation  of  hardware/computer  programs  or  any 
of  its  discrete  portions,  which  satisfies  an  end-use  function  and  is 
designated  by  the  Government  for  configuration  management.  CIs  may  vary 
widely  in  complexity,  size,  and  type,  from  an  aircraft,  electronic,  or 
ship  system  to  a test  meter  or  round  of  ammunition  (abbreviated,  from 
AFR  65-3). 

Critical  Design  Review  (CDR).  A formal  technical  review  of  the  design  as 
depicted  by  the  specification  and  flow  diagrams,  sufficiently  detailed 
to  enable  the  programmer  to  code  and  to  assure  that  design  require- 
ments have  been  met  before  beginning  coding. 

Development  Specification.  A document  applicable  to  a Cl  which  states  all 
necessary  requirements  in  terms  of  performance,  interface,  design 
constraints,  functional  characteristics,  and  the  tests  required  to 
demonstrate  achievement  of  those  requirements  (see  MIL-STD-490A  for 
subtypes  of  Development  Specifications). 
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Formal  Qualification  Review  (FQR).  The  test,  inspection,  or  analytical  pro- 
cess by  which  products  at  the  end  item  or  cri tical -i tern  level  are  verified 
to  have  met  specific  procuring  activity  contractual  performance  require- 
ments (specification  or  equivalent). 

Formal  Qualification  Tests  (FQT).  Formal  tests  oriented  toward  testing  of  the 
functional  and  performance  characteristics  of  the  CPCI,  normally  using 
operationally  configured  equipment  at  the  System  DT&E  site  prior  to  the 
beginning  of  System  DT&E. 

Full-Scale  Development  Phase.  I he  period  when  the  system/equipment  and  the 
principal  items  necessary  for  its  support  are  designed,  fabricated, 
tested,  and  evaluated.  The  intended  output  is,  as  a minimum  a preproduc- 
tion system  which  closely  approximates  the  final  product,  the  documenta- 
tion necessary  to  enter  the  production  phase,  and  the  test  results  which 
demonstrate  that  the  production  product  will  meet  stated  requirements 
(DODI  5000.1/AFR  800-2). 

Functional  Configuration  Audit  (FCA).  A formal  audit  to  validate  that  the 
development  of  a Cl  has  been  completed  satisfactorily  and  that  the  Cl  has 
achieved  the  performance  and  functional  characteristics  specified  in  the 
functional  or  allocated  configuration  identification. 

Functional  Flow  Analysis.  An  iterative  process  used  in  identifying,  analyzing, 
detailing,  and  sequencing  the  system  functions  thru  the  use  of  functional 
flow  charts.  The  term  function  being  an  operation  the  system  must  perform 
to  fulfill  its  intended  mission. 

Qualification.  The  entire  process  by  which  products  are  obtained  from  manu- 
facturers or  distributors,  examined,  and  tested. 

Physical  Configuration  Audit  (PCA).  A formal  examination  of  the  technical 
documentation  (specification,  handbooks,  and  manuals)  to  determine  their 
compatibility  with  the  qualified  CPCI. 

Preliminary  Design  Review  (PDR).  A formal  review  of  the  preliminary  design  of 
a Cl  to  (1)  evaluate  technical  progress,  (2)  determine  its  compatibility 
with  the  requirements  of  the  Cl  Development  Specification,  and  (3) 
establish  the  existance  and  compatibility  of  the  physical  and  functional 
interfaces  among  Cl  equipment,  facilities,  computer  programs,  and 
personnel . 

Preliminary  Qualification  Tests  (PQT).  Formal  tests  oriented  primarily  toward 
verifying  portions  of  the  CPCI  prior  to  integrated  testing/ formal  qualifi- 
cation tests  of  the  complete  CPCI.  These  tests  will  typically  be  conducted 
at  the  contractor's  design  and  development  facilities. 

Product  Specification.  For  a CPCI,  a document  which  provides  a detailed 
technical  description  of  the  CPCI  [MIL-STD-490  and  MIL-STD-483  (USAF)]. 

System  Design  Review  (SDR).  A design  review  conducted  to  evaluate  the  optimi- 
zation,  correlation,  completeness,  and  risk  associated  with  the  allocated 
technical  requirements. 


System  Requirements  Review  (SRR).  A system  engineering  review  to  ascertain 
the  adequacy  of  the  contractor's  efforts  in  defining  system  requirements. 

It  will  be  conducted  when  a significant  portion  of  the  system  functional 
requirements  has  been  established. 

System  Segment  Specification.  A specification  of  the  same  format  as  a System 
Specification,  identifying  a discrete  package  of  system  performance  require- 
ments, functional  interfaces  and  configuration  items  contracted  to  one  con- 
tractor or  assigned  to  one  Government  organization  directly  resoonsible 
for  the  procuring  activity  for  that  part  of  the  system's  performance. 

System  Specification.  A document  which  states  all  the  necessary  technical 
and  mission  requirements  in  terms  of  performance,  allocates  requirements 
to  functional  areas  (or  CIs),  defines  the  interfaces  between  or  among 
the  functional  areas  (or  CIs),  and  includes  the  test  provisions  to  assure 
the  achievement  of  all  requirements. 

Validation  Phase.  The  overall  objective  of  the  Validation  Phase  is  to  deter- 
mine whether  to  proceed  with  Full-Scale  Development.  The  ultimate  goal 
of  the  Validation  Phase,  where  development  is  to  be  performed  by  a con- 
tractor, is  to  establish  firm  and  realistic  performance  specifications 
(Allocated  Baseline),  which  meet  the  operational  and  support  requirements. 

Val idation.  Validation  is  the  process  by  which  the  contractor  tests  operating 
and  maintenance  procedures  for  technical  accuracy  and  adequacy. 

Verification.  Verification  is  the  process  by  which  operating  and  maintenance 
procedures  are  tested  and  proven  under  Air  Force  jurisdiction. 
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ACRONYMS  AND  ABBREVIATIONS 


AFR  Air  Force  Regulation 

3 

C Command,  Control,  and  Communication 

CDR  Critical  Design  Review 

Cl  Configuration  Item 

CPCI  Computer  Program  Configuration  Item 

CPDP  Computer  Program  Development  Plan 

CPT&E  Computer  Program  Test  and  Evaluation 

DT&E  Development  Test  and  Evaluation 

ECP  Engineering  Change  Proposal 

FCA  Functional  Configuration  Audit 

FQR  Formal  Qualification  Review 

FQT  Formal  Qual i fication  Test 

PCA  Physical  Configuration  Audit 

PCD  Program  Contracting  Officer 

PDR  Preliminary  Design  Review 

PO  Program  Office 

PQT  Preliminary  Qual  i'fi  cat  ion  Test 

QQPRI  Quantitative  and  Qualitative  Personnel  Requirements  Information 

RSS  Regul ations,  Speci fications , and  Standards 

SCN  System  Change  Notice 

SD  Software  Director 

SDR  System  Design  Review 

SE/TD  System  Engineering/Technical  Direction 

SRR  System  Requirements  Review 

WBS  Work  Breakdown  Structure 
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